January 2019

This document sets out the assessment criteria and working methods of the main a nd sub-

panels for the Research Excellence Framework (REF) 2021.

Panel criteria and

working methods

Department for the Economy, Northern Ireland

Higher Education Funding Council for Wales

Research England

Scottish Funding Council

 
1 
 
Contents 
Contents ....................................................................................................................................... 1
                         
Executive summary ...................................................................................................................... 2
                         
Introduction ................................................................................................................................... 4
                         
Part 1: Overview ........................................................................................................................... 5
                         
Part 2: Unit of assessment descriptors ...................................................................................... 
                        12 
                        
Part 3: Assessment criteria ........................................................................................................ 
                        37 
                        
Part 4: Panel procedures............................................................................................................ 
                        83 
                        
Part 5: Panel working methods .................................................................................................. 
                        84 
                        
Annex A: Examples of impacts and indicators ........................................................................... 
                        93 
                        
Annex B: Summary of additional information about outputs .................................................... 
                        110 
                        
Annex C: Main Panel D – output types & submission guidance.............................................. 
                        113 
                        
Annex D: Managing conflicts of interest ................................................................................... 
                        120 
                        
Annex E: Confidentiality and information security agreement for REF 2021 panels
                        ............... 124 
                    
Annex F: List of abbreviations .................................................................................................. 
                        127

2

REF 2021: Panel criteria and working methods

To

Heads of higher education institutions in the UK

Of interest to those

responsible for

Research

Reference

REF 2019/02

Publication date

January 2019

Enquiries from staff at UK

higher education

institutions

Email your institutional REF contact. (These are listed at www.ref.ac.uk,

under Contact.)

Other enquiries

Hannah Daisley, tel 0117 931 7486, email info@ref.ac.uk

Executive summary

Purpose

1. This document sets out the assessment criteria and working method s of the main and sub-

panels for the Research Excellence Framework (REF) 2021.

2. This document, taken together with the ‘Guidance on submiss ions’ (REF 2019/01), provides a

comprehensive description of the information required in submissions to the REF, and how the REF

panels will assess submissions.

Key points

3. The REF is a process of expert review. Expert sub-panels f or each of 34 units of assessment

(UOA) will carry out the assessment, working under the leadership and guidance of four main

panels.

4. UK higher education institutions (HEIs) will be invited to make submissions by 27 November

2020. The REF main and sub-panels will assess submissions during 2021, and results w ill be

published in December 2021. The results will inform the allocation of research funding by the UK

higher education (HE) funding bodies, from 2022–23.

5. This document sets out a combined statement of criteria and working methods across the four

main panels. Supplementary criteria are set out for each main panel where applicable. The main

panels’ supplementary criteria are intended to be read alongside the combined criteria, and do not

replace it.

Action required

6. This document is for information and to guide institutions in preparing and collecting data for

inclusion in REF submissions. No action is required by HEIs at this stage.

3

Further information

7. For further information about the REF see www.ref.ac.uk .

8. Staff at UK HEIs should direct any queries to their institutional REF cont act. Contact details for

each institution are listed at www.ref.ac.uk, under Contact.

9. Other enquiries should be addressed to info@ref.ac.uk.

4

Introduction

10. This document sets out the assessment criteria and working method s of the main and sub-

panels for REF 2021.

11. The criteria have been developed as a combined statement across the four main panels. This

responds to feedback following REF 2014 that there was further scope for increase d consistency in

the criteria, and incorporates advice from the main and sub-panels on where further alignm ent

across the criteria could be achieved. Supplementary criteria are set out for each main panel where

applicable. The main panels’ supplementary criteria are intended to be read alongside t he combined

criteria, and do not replace it.

12. In early 2018, the REF team invited the four main panels to develop the ir criteria and working

methods, with input from their sub-panels. The REF team provided guidance to the pane ls on

developing their criteria. The ‘Guidance to panels’ is available at www.ref.ac.u k, under Publications.

This invited the main panels to develop a combined set of criteria and working methods, with

supplementary criteria for each main panel and its group of sub-panels where applicable. Th e main

panels were invited to agree differences in approaches for particular sub-panels within their remit

only where justified by differences in the nature of research in those disciplines.

13. The assessment criteria and working methods are set out in this document as follows:

 Part 1 provides an overview of the REF and the expert panels.

 Part 2 sets out the descriptor for each of the 34 units of assessment (UOAs).

 Part 3 gives further details of the assessment criteria to be employed by the four main panels

and their sub-panels, including the combined criteria and, where applicable, any

supplementary criteria provided by any main panel.

 Part 4 sets out the generic panel procedures in place across the main and sub-panels.

 Part 5 sets out the combined working methods of the main and sub-panels.

14. This document should be read alongside REF 2019/01 ‘Guidan ce on submissions’ (hereafter

‘Guidance on submissions’), which provides an overview of the REF a ssessment framework and

guidance to institutions on preparing their submissions, including the data requirement s and

definitions that apply.

15. Together, the two documents will give a comprehensive description o f the information required

in submissions to the REF, and how the REF panels will assess submissions. We will issue

supplements to the guidance at later dates to clarify points of detail regarding submissions. Panels

will not be permitted to depart from the final criteria once published, other than in exceptional

circumstances that cannot be accommodated within the published framework. In such cas es, we will

publish the reason and details of the change as an amendment.

5

Part 1: Overview

16. The REF is the system for assessing the quality of research in HEIs in the UK. It was first

conducted in 2014, and replaced the previous Research Assessment Exercise (RAE) .

17. The purpose of the REF, the general principles governing its conduct, and an overview of the

REF framework are set out in Part 1 of ‘Guidance on submissions’.

Submissions and units of assessment

18. Institutions will be invited to make submissions by 27 Nove mber 2020. There are 34 UOAs,

listed in Annex D of ‘Guidance on submissions’. Part 2 of this document provides descriptors of each

UOA. Each submission must contain, in summary:

a. REF1a/b: Information on all staff in post on the census dat e, 31 July 2020, with

significant responsibility for research; and information about former staff to whom

submitted outputs are attributed.

b. REF2: Details of assessable outputs that the submitt ed unit has produced during the

publication period (1 January 2014 to 31 December 2020). The total number of outputs

must equal 2.5 times the summed full-time equivalent (FTE) of the unit’s sub mitted staff.

Rounding to the nearest whole number will be applied to give a whole number of outputs

for submission.

c. REF3: Case studies describing specific examples of impacts ach ieved during the

assessment period (1 August 2013 to 31 July 2020), underpinned by excellent researc h

in the period 1 January 2000 to 31 December 2020.

d. REF4a/b/c: Data about research doctoral degrees awarded, resear ch income and

income-in-kind related to the period 1 August 2013 to 31 July 2020.

e. REF5a/b: An institutional-level environment statement, and a completed template

describing the submitted unit’s research and impact environment, drawing on

quantitative indicators as appropriate, and related to the period 1 August 2013 to 31 July

2020.

f. REF6a/b: Information on staff circumstances, where cla iming a reduction or removal of

the requirement to submit a minimum of one output per member of staff submitt ed.

19. The generic eligibility definitions and data requirements set out in ‘Guidance on submissions’

apply to all submissions.

6

Expert panels

20. The REF will be a process of expert review, with an expert panel assessing submissions made

by HEIs in each of the 34 UOAs. The sub-panels will work under the leadership and guidanc e of four

main panels.

21. In brief, the four main panels are responsible for:

 developing the panel criteria and working methods

 overseeing calibration

 ensuring adherence to the published procedures and consistent application o f the overall

assessment standards by the sub-panels

 working with the advisory panels for equality and diversity and interdiscip linary research

(IDR), and advising the REF team on assessment processes

 signing off the outcomes of the assessment.

22. The sub-panels are responsible for:

 contributing to the criteria and working methods

 working within the agreed criteria in assessing each submission made in its UOA

 recommending the outcomes for each submission to the main panel.

23. The roles and responsibilities of the main and sub-panels are described fu lly in ‘Roles and

recruitment of the expert panels’ (REF 2017/03).

24. The main and sub-panels will undertake their roles within the common frame work for

assessment set out in ‘Guidance on submissions’ and the combined statement of criteria and

working methods (Parts 2 to 5 of this document). Part 3 of this document also sets out any

supplementary criteria that each of the main panels and its sub-panels will employ when assessing

submissions.

25. The main and sub-panels are appointed by the four UK funding bodies through an open

process of nominations, as described in REF 2017/03. The appointment of the expert pan els will be

undertaken in two stages for REF 2021:

a. Criteria phase: sufficient members were appointed in 2018 to ensure each sub-panel

has appropriate expertise (including in IR and the wider use of research) for contr ibuting

to the criteria development. This is a smaller group of members than the volume

required to undertake the assessment, meaning that further full sub-panel members

(and impact and output assessors) will join the sub-panels for the assessment phase.

7

The members joining at the criteria phase will input into the panel criteria development,

and will contribute to the assessment of all elements of submissions (outputs, impact

and environment) in 2020–2021.

b. Assessment phase: additional sub-panel members will be appointed for the

assessment phase of the exercise, to ensure the sub-panel has an appropriate v olume

of members who will contribute to the assessment of all elements of submiss ions.

26. The membership of the panels appointed for the criteria phase is at www.ref.ac.uk, under

‘Expert Panels’. Following advice from main and sub-panel chairs, where suitable nominations were

received the funding bodies have appointed a proportion of the members who will jo in the sub-

panels for the assessment phase. These members will take up their panel rol es in 2020, and are

also detailed on the REF website.

Appointment of additional sub-panel members and assessors

27. We will make further appointments to the panels of both sub-panel members and assessors in

advance of the assessment year. This is to ensure the sub-panels have an appropriat e breadth of

expertise and number of panel members necessary for the assessment phase. We also expect to

make further appointments of members with the role of interdisciplinary adv iser in the assessment

phase. This role is described in REF 2017/03 (Box 1).

28. A survey of institutions’ submission intentions will be carried o ut in 2019, which will inform

additional panel membership appointments, including for both further full sub-panel memb ers as well

as output and impact assessors.

29. Further members and assessors will be appointed by the chief executives (or equivalent) of

the four UK funding bodies, following recommendations from main and sub- panel chairs, made from

nominated individuals. These will either be individuals with appropriate expertise w ho were

nominated in the initial round of nominations in 2017 (see REF 2017/03), or additional no minations

that the REF team will invite in a further round in 2020. This will include nominations for those with

expertise in the use or benefits of research across the private, public and third sectors.

30. Where further nominations are invited, the REF team will again ask nominating bodies to

provide information on how equality and diversity considerations were taken a ccount of during the

nominations process. The Equality and Diversity Advisory Panel (EDAP) have reviewed the

templates provided by nominating bodies in the 2017 round of nominations, and will produce a report

summarising good practice identified in the nominations process. Nominating bod ies will be invited

to consider this report when making nominations in 2020. In recommending further pane l members

and assessors, sub-panel chairs will give consideration to enhancing the extent t o which the overall

body of members reflects the diversity of the research community. The REF tea m are currently

undertaking an analysis of the representativeness of panel membership, which will be able to further

inform chairs’ considerations in this regard.

8

Assessors

31. Assessors appointed to the sub-panels will undertake either one of the following roles:

a. To assess the impact element of submissions and develop the impact sub-profiles,

alongside sub-panel members. These will be people with professional experience o f

making use of, applying or benefiting from academic research.

b. To assess research outputs and develop the output sub-profiles, alongside existing

panel members. These will be practising researchers with relevant expertise.

32. Assessors will play a full and equal role to sub-panel members in developi ng the sub-profiles

for either the impact or outputs element of the assessment. They will be fully briefed, take part in

calibration exercises and attend panel meetings at which the relevant aspects of su bmissions

(outputs or impact) are discussed.

Appointment process

33. Main and sub-panel chairs’ recommendations for further membersh ip will be guided by the

principle of ensuring that sub-panels have access to appropriate expertise to reach r obust and valid

judgements with regard to submitted material. Appointments will be made as follows:

a. In 2019, the REF team will survey institutions about the volume and nature of work that

they intend to submit to the REF. The survey data will provide information to the REF

team and the panels about how the changes to the submission process in REF 2021

may affect the volume and range of work submitted, compared with previous exerc ises.

b. In early 2020, the main and sub-panels will consider the breadth and depth of expertise

of each sub-panel’s current membership, in the light of institutions’ submission

intentions. Each sub-panel will seek to identify:

 where the membership required to assess submissions should be ex panded to

ensure appropriate expertise in accordance with the anticipated volume of

submissions

 where additional IDR advisers may be required to support and advise the sub-panel

on the assessment of IDR outputs

 areas where additional expertise would be required to assess the range a nd volume

of outputs indicated in the survey responses

 areas where additional user expertise would be required to assess th e range of

impacts indicated in the survey responses.

9

c. Where a clear gap in the expertise of a sub-panel required to assess either outputs or

impact has been identified during the criteria development and consultation phase,

additional appointments may be made during 2019.

34. Before recommending the appointment of further members and assessors, sub-panel chairs

will discuss the recommendations with their main panels. The following issues will be considered

across each main panel:

 the identified expertise requirements in the context of the curre nt panel membership’s

breadth and volume of expertise

 the overall size of the sub-panel in the context of anticipated sub missions

 the potential for interdisciplinary advisers to work across sub- panels

 the need to ensure that impact case studies are given fair considerat ion, with the intention of

ensuring that there is sufficient user expertise to review the range of likely imp act case

studies that will be submitted

 where assessors are recommended:

o whether there is a sufficient body of activity requiring an additional assessor appointment

o whether serious workload issues or conflicts of interest for existing pa nel members have

been identified, requiring an additional assessor for a particular subject area

o the potential for individual assessors to be appointed to two sub-panels, where there is a

significant overlapping body of work expected (and, if appropriate, the potential to

appoint existing user members to also act as assessors for other sub-panels).

35. Once appointed at each stage, the names of sub-panel members and assess ors will be

published on www.ref.ac.uk alongside the current panel membership. Assess ors will be eligible to

receive fees and expenses on the same basis as panel members.

36. As stated in REF 2017/03 (paragraph 67), as the REF progresses, ma in or sub-panels may

recommend to the funding bodies the appointment of a small number of members or assessors in

addition to the members already appointed and/or the members and assessors to be appointed

through the processes outlined above, to provide further expertise where this is desirable an d in

accordance with the criteria for appointments.

Assessing submissions

37. As with the previous REF and RAEs, the assessment process is based on expert review. Each

sub-panel will examine the submissions made in its UOA, taking into account all the evidence

10

presented. Each sub-panel will use its professional collective judgement to form an overall view

about each submission and recommend to the main panel an overall quality profile to be awarded to

each submission made in its UOA. The process is objective and evidence based but is not and

cannot be purely algorithmic.

38. The primary outcome of the panels’ work will be an overall quality profile awarded to each

submission, to be published in December 2021. An example overall quality profile is in Annex B of

‘Guidance on submissions’, and further details about the published out comes and feedback that

panels will produce are in paragraphs 34 to 43 of that document.

39. In forming their overall quality judgements, the sub-panels will assess three distinct elements

of each submission – outputs, impact and environment – against the following generic criteria:

a. Outputs: The sub-panels will assess the quality of sub mitted research outputs in terms

of their ‘originality, significance and rigour’, with reference to internationa l research

quality standards. This element will carry a weighting of 60 per cent in the overall

outcome awarded to each submission.

b. Impact: The sub-panels will assess the ‘reach and signifi cance’ of impacts on the

economy, society, culture, public policy or services, health, the environment or quality of

life that were underpinned by excellent research conducted in the submitted unit. This

element will carry a weighting of 25 per cent.

c. Environment: The sub-panels will assess the research environment in terms of its

‘vitality and sustainability’, including the approach to enabling impact from its researc h,

and its contribution to the vitality and sustainability of the wider discipline or research

base. This element will carry a weighting of 15 per cent.

40. The generic definitions of the starred quality levels in the overall qua lity profile in each of the

three sub-profiles – outputs, impact and environment – are in Annex A of ‘Guidance on

submissions’. All sub-panels will apply these generic assessment criteria, level definitions and

weightings for each element, in forming the overall quality profiles to recommend t o their main panel.

41. In Part 3 of this document, the main panels’ criteria stat ement provides a descriptive account

of these generic assessment criteria, and of the starred level definitions for out puts. These are

provided to inform their subject communities on how the panels will apply the criteria and definitions

in making their judgements. These descriptive accounts should be read alongside, but do not

replace, the generic definitions.

Outputs

42. As set out in ‘Guidance on submissions’, each submission must include a set number of items

of research output, equal to 2.5 times the combined FTE of Category A submitted staf f included in

the submission. Rounding to the nearest whole number will be applied to give a whole number of

11

outputs for submission. The total number of outputs for return may be adjusted, as appropriate, to

take account of staff circumstances (see ‘Guidance on submissions’, paragraphs 151 to 201).

43. An underpinning principle of the REF is that for each discipline all types of research and all

forms of research output shall be assessed on a fair and equal basis. Panels hav e been instructed to

define criteria and adopt assessment processes that enable them to recognise, and tr eat on an

equal footing, excellence in research across the spectrum of applied, practice, basic and strategic

research, wherever that research is conducted; and for identifying excellence in different forms of

research endeavour including interdisciplinary and collaborative research, while attac hing no greater

weight to one form over another.

44. ‘Guidance on submissions’ (Annex C) sets out the generic definition o f research and its Annex

K provides a glossary of the different output types that may be submitted, and is intended to highlight

the diversity of research outputs that may be submitted in any UOA. Any assessab le form of output

that embodies research is eligible for assessment.

Impact

45. ‘Guidance on submissions’ sets out the generic submission r equirements in relation to impact,

including the number of case studies required in each submission (paragraph 309 ), the eligibility

criteria for impact case studies (paragraphs 311 to 313), and a template an d guidance on completing

impact case studies (Annex G).

46. The generic definition of impact for the REF given in ‘ Guidance on submissions’ (Annex C) is

broad, and any impact that meets this definition is eligible for assessment, in any UOA. The panel

criteria statement in Part 3 and examples in Annex A in this document provide some furt her

descriptions of the diversity of impacts that may apply in their UOAs. These ar e provided to inform

their subject communities: they should be read alongside, but do not repla ce, the generic definition

in ‘Guidance on submissions’.

47. The main panels’ criteria statement in Part 3, and the examples in Annex A, provide guidance

on the forms of evidence that would be appropriate for submissions to include in impact case studies

(REF3). The statement in Part 3 also states how the panels will assure that the quality of research

that underpins impact case studies is equivalent to at least two-star quality.

Environment

48. ‘Guidance on submissions’ sets out the generic requirements for the environment element of

submissions, which comprise:

a. standard data on research doctoral degrees awarded, research income an d research

income-in-kind (REF4a/b/c)

b. a completed institutional-level environment statement (REF5a)

c. a completed environment template (REF5b).

12

49. In Part 3, the panel criteria provide guidance on the forms of evidence that would be

appropriate for submissions to include in the environment template (REF5b), inc luding any

quantitative indicators that should be provided within REF5b, where applicable. The template for

REF5b is set out in Annex I of the ‘Guidance on submissions’.

50. REF panels will form an environment sub-profile by assessing the informatio n submitted in

REF5b, informed by the data submitted in REF4a/b/c and REF5a. When t he REF team provides

submissions to sub-panels, we will supply a standard analysis of the quantitativ e data submitted in

REF4a/b/c, in respect of each submission in that UOA, and aggregated for all sub missions in that

UOA (see ‘Guidance on submissions’, Annex J). Panels will consider these data within the context of

the information provided in REF5b, and within the context of the disciplines concern ed. In Part 3, the

panels’ criteria statement indicates how the data analyses will be used in informing t he assessment

of the research environment.

Part 2: Unit of assessment descriptors

51. Details of the assessment of interdisciplinary work and work that crosses UOA boundaries,

including pedagogic research, are provided in paragraphs 166 to 177.

Main Panel A: UOAs 1–6

Introduction

52. The UOAs within Main Panel A’s remit cover research into t he practices, services, policies,

education and underpinning science relevant to these disciplines, and associated m ethodological

and theoretical advancement. The UOAs cover a full spectrum of research approaches, ranging from

qualitative to quantitative, as well as theoretical and mixed method studies. This inc ludes IDR and

research that informs these areas from a range of stakeholders’ perspectives, includin g research

users and service users.

53. Where it is relevant to the UOAs, submissions will be welcome whet her the context is local,

national or international, including work carried out in relation to developing countries.

54. The main panel encourages institutions to structure their submissio ns using research groups,

noting that there is no expectation that submissions will necessarily comprise a single coher ent body

of research. Where submissions are structured using research groups, the sub-panels’ written

qualitative feedback to institutions may highlight individual research groups of particular note.

Unit of assessment descriptors and boundaries

Unit of Assessment 1: Clinical Medicine

55. The UOA includes research into all aspects of clinical medicine and its cognate sub-disciplines

except for bodies of research more explicitly linked to UOA 2 (Public Health, H ealth Services and

13

Primary Care), UOA 3 (Allied Health Professions, Dentistry, Nursing and Phar macy), UOA 4

(Psychology, Psychiatry and Neuroscience) and UOA 5 (Biological Sciences).

56. The sub-panel expects submissions that demonstrate integrated strateg ies relating to all

aspects of medical research. Submissions may cover the full range of researc h related to medicine,

from basic underpinning studies through experimental medicine to clinical trials. In v iew of the

breadth of research covered by this UOA, the sub-panel expects some degree of overla p with UOA

4 (Psychology, Psychiatry and Neuroscience) in the fields of neurology and with UO A 5 (Biological

Sciences) in the area of basic biological sciences underpinning medical research.

Unit of Assessment 2: Public Health, Health Services and Primary Care

57. The UOA includes research into all aspects of public health, health s ervices and/or primary

care and all their cognate disciplines. The research may be applied, theoretical or methodological

research from relevant health or healthcare disciplines.

58. The sub-panel expects submissions in this UOA from all areas of pub lic health and

epidemiology (from aetiology to intervention), health services and primary care, including c linical

trials, health social sciences, health policy research and healthcare management, and fro m other

related disciplines having a relevance to the research covered by the UOA. It recognises t he breadth

and diverse range of single, multidisciplinary and/or multi-professional research across pub lic health,

health services and primary care.

Unit of Assessment 3: Allied Health Professions, Dentistry, Nursing and

Pharmacy

59. The UOA includes research into all aspects of the disciplines of alli ed health professions,

dentistry, nursing, midwifery and pharmacy. Its boundaries include research in under pinning science,

laboratory-based work, applied clinical research, healthcare technologies, and research into public

health, social care and health promotion. Research into psychosocial, philosophical an d ethical

aspects of healthcare, as well as education, policy and methodology relevant to these d isciplines, is

also included. It is anticipated that such work will use qualitative, quantitative and m ixed methods, as

well as theoretical approaches.

60. For allied health professions, submitted research is expected to underpi n clinical practice and

policy development and implementation, and includes research in biomedical and nutrition al

sciences, dietetics, biology of health and disease, vision sciences, optometry, orthoptic s, osteopathy,

operating department practitioners, diagnostic imaging, therapeutic radiography, audiology, podiatry,

occupational therapy, physiotherapy, speech and language therapy, clinical linguistics, para medics,

prosthetics/orthotics, music therapy, drama therapy and arts therapy. For dentistry, it includes

research in basic and applied dental, oral and craniofacial sciences encompassing all t he related

clinical disciplines, primary dental care, biomaterials sciences relevant to oral and craniofacial

science, and other such sciences relevant to dentistry. For nursing and midwifery, it includes

specialist, community and public health nursing, and all the contexts within which they operate. For

pharmacy, it includes all aspects of the design, synthesis, formulation, action and use of

14

pharmaceuticals (including biological and nutraceuticals), to include medicinal che mistry,

pharmaceutics, pharmacology, clinical pharmacy, underlying biomedical science, and the practice of

pharmacy.

61. Submissions may cover the full translational range of research, fro m basic underpinning

studies through to implementation research. Bodies of research more explicitly linked to UOA 1

(Clinical Medicine), UOA 2 (Public Health, Health Services and Primary Care) , UOA 4 (Psychology,

Psychiatry and Neuroscience), UOA 5 (Biological Sciences) and UOA 6 (Agriculture, Food and

Veterinary Sciences) should be submitted to those panels and such outputs su bmitted to UOA 3 will

be cross-referred.

Unit of Assessment 4: Psychology, Psychiatry and Neuroscience

62. The UOA expects submissions in all areas of psychology, psy chiatry and neuroscience.

63. This includes all areas of psychological research with humans and animals. It covers

quantitative and qualitative approaches to typical and atypical populations in all settings at the

cultural, societal, group and individual levels, across the full range of areas of psychological practice

and outside of formalised settings, and includes all aspects of psychological e xperience. For

psychiatry, it covers research on all aspects of the study of mental disorders. This includ es their

aetiology, epidemiology, mechanisms and consequences, as well as pharmacologica l and non-

pharmacological treatments, systems of care and mental health policy. For neurosc ience it covers all

aspects of research from the molecular through to whole-system behavioural level, geneti cs and

varieties of imaging, and both neurodevelopmental and adult work. This includ es research on the

understanding and treatment of all types of neurological and neurosurgical conditions, i ncluding

those related to neurodegeneration and neurodevelopment.

64. The remit of the sub-panel is broad, covering submissions with the potential to transform

research into practice as well as those reporting theoretical and methodological adv ances in basic

research. Overlap is expected with UOA 1 (Clinical Medicine), UOA 2 (Public Hea lth, Health

Services and Primary Care), UOA 3 (Allied Health Professions, Dentistry, Nursing and Pharmacy),

UOA 5 (Biological Sciences), UOA 8 (Chemistry), UOA 9 (Physics), UOA 11 (Computer Science and

Informatics), UOA 12 (Engineering), UOA 17 (Business and Management St udies), UOA 20 (Social

Work and Social Policy), UOA 21 (Sociology), UOA 23 (Education), UOA 24 (Sports and Exercise

Sciences, Leisure and Tourism), UOA 26 (Modern Language and Linguistics) and UOA 30

(Philosophy).

Unit of Assessment 5: Biological Sciences

65. The UOA includes research into all aspects of biological and biomedical sciences that

encompasses the full spectrum of the fundamental and applied biology of all organisms, at all levels

of organisation from the molecular to the ecosystem, employing a diversity of approaches inc luding

experimental, theoretical, computational and mathematical. The UOA also cov ers all aspects of the

biomedical sciences, including biochemistry, physiology, pharmacology and anato my at the genetic,

15

molecular, cellular, organ system and whole-organism level. It includes work relevant to the nervous

and cardiovascular systems at all levels of enquiry.

66. Submissions may include work which is on the boundaries of other UOAs in Main Panel A,

such as: UOA 1 (Clinical Medicine), UOA 3 (Allied Health Professions, Dentistry , Nursing and

Pharmacy), UOA 4 (Psychology, Psychiatry and Neuroscience) and UOA 6 ( Agricult ure, Food and

Veterinary Sciences); as well as UOAs in other main panels, such as: UOA 7 (Earth Sy stems and

Environmental Sciences), UOA 8 (Chemistry), UOA 9 (Physics), UOA 10 (Mathematical Sciences),

UOA 11 (Computer Science and Informatics), UOA 14 (Geography and Environmental St udies) and

UOA 24 (Sport and Exercise Sciences, Leisure and Tourism).

Unit of Assessment 6: Agriculture, Food and Veterinary Sciences

67. The UOA includes research into all aspects of the agricultural, foo d and veterinary sciences,

including basic through to applied research, and interdisciplinary research with sign ificant content in

any of these areas of science.

68. The sub-panel expects submissions in this UOA from all areas of relevant science.

a. In agricultural science this includes the scientific exploration of all aspects of agriculture,

horticulture, forestry, aquaculture and marine, including the sciences underpinning and

impacted by them, and the emerging technologies that support these industries with

particular reference to the combination of economic, environmental and social

disciplines into defining optimal systems of land use, the development of the bio-

economy and integrated approaches to farming.

b. For veterinary science, submissions may cover the full range of research r elated to

veterinary medicine and surgery. These include studies ranging from basic underpinn ing

research through to clinical, applied and social science. Submissions may include

research relevant to normal and abnormal function of animals, their health, behaviour,

welfare, nutrition and productivity, as well as their role in human health, environment

and society, or as models for human diseases.

c. For food, this includes submissions of primary relevance to food sc ience and technology

(including chemistry, physics, microbiology, engineering and processing), human

nutrition, diet and health, food biotechnology, food safety, packaging, sensory science,

food consumer science, and food security, sustainability and environmental aspects.

69. In view of the breadth of research covered by this UOA, the sub-panel expects some degree

of overlap with underpinning sciences research which may be presented to all panels.

16

Main Panel B: UOAs 7–12

Introduction

70. The six sub-panels that fall within Main Panel B invite submissions in UOAs 7 to 12 as set out

in the following paragraphs.

71. Institutions should make submissions to the most appropriate UO A. Sub-panels will utilise

cross-referral and interdisciplinary research processes where appropriate.

72. Main Panel B recognises that work in materials science takes places in several of its

disciplines and as a result crosses the boundaries of UOAs 7, 8, 9 or 12.

73. HEIs may choose to associate outputs with research groups if they have used these research

groups to structure their environment submission. This is not a mandatory field and neither the

presence nor absence of research groups is assumed.

Unit of assessment descriptors and boundaries

Unit of Assessment 7: Earth Systems and Environmental Sciences

74. The UOA includes Earth, environmental and planetary sciences, including: geophysics;

geochemistry; palaeontology; geology; mineral physics; evolution of planetary atmospheres,

surfaces and interiors; Earth surface processes; the physics, chemistry and biology of the

environment, including ecology and conservation; atmospheric, marine, freshw ater, terrestrial and

soil sciences; innovative measurement systems and data analysis; global change; natur al resources;

natural hazards; pollution; and environmental management and impact.

75. The sub-panel expects submissions in this UOA from all areas of Earth systems and

environmental sciences, as defined above, and expects that the majority of the research activity

submitted will have made a direct contribution to the UOA as characterised in the UOA descriptor. It

recognises, however, the increasingly interdisciplinary nature of research, and expects that

submissions may contain work that contributes to this UOA and other cognate disc iplines. It is

expected, however, that submissions will be made to the UOA where there is the most appropriate

expertise to assess the body of work as a whole.

Unit of Assessment 8: Chemistry

76. The UOA includes all areas of experimental and theoretical che mistry, including appropriate

areas of pharmacy, chemical engineering and materials science, where the research is primarily

concerned with chemical aspects rather than clinical or engineering.

77. The sub-panel expects submissions in this UOA from all areas of chemistry, as defined above,

and expects that the majority of the research activity submitted will have made a direct contribution

to the UOA as characterised in the UOA descriptor. It recognises and welcomes, how ever, the

increasingly interdisciplinary nature of research, and expects that submissions may contain work that

contributes to this UOA and other disciplines, including those which have boun daries with this UOA,

17

such as UOA 5 (Biological Sciences), UOA 7 (Earth Systems and Environment al Sciences), UOA 9

(Physics) and other cognate disciplines.

Unit of Assessment 9: Physics

78. The UOA includes all areas of physics encompassing, but not limited to, theoretical,

computational and experimental studies of: quantum physics; atomic, molecular an d optical physics;

plasma physics; fusion and energy; particle physics; nuclear physics; surface and interface physics;

condensed matter, materials and soft matter physics; biophysics; semiconductors, nanoscale

physics, lasers, optoelectronics and photonics; magnetism, superconductivity and quantum fluids;

fluid dynamics; statistical mechanics, chaotic and nonlinear systems; astrono my and astrophysics,

planetary and atmospheric physics; space physics; cosmology and relativity; medi cal physics;

applied physics; chemical physics; and instrumentation.

79. The sub-panel expects submissions in this UOA from all areas of physics, as defined above,

and expects that the majority of the research activity submitted will have made a direct contribution

to the UOA as characterised in the UOA descriptor. It recognises and welcomes, however, the

increasingly interdisciplinary nature of research, and expects that submissions may contain work that

contributes to this UOA and other cognate disciplines.

Unit of Assessment 10: Mathematical Sciences

80. The UOA includes pure and applied mathematics, statistics and operatio nal research,

including the development and application of these areas in the study of biological, physical and

social sciences, commerce, engineering, finance, government, health, industry, information science,

medicine and elsewhere.

81. It therefore includes: algebra; analysis; category theory; combinatorics; complexity theory;

continuum mechanics and magnetohydrodynamics; differential equations; dynamical sy stems and

ergodic theory; environmental, financial, geophysical and industrial mathematics; geometry;

integrable systems; mathematical biology; mathematical logic; mathematical methods; mathematical

aspects of operational research, including optimisation and stochastic modelling; mat hematical

physics; number theory; numerical analysis and scientific computing; operator theory and operator

algebras; probability; statistics such as experimental design, mathematical statistics, st atistical

computing and contributions to data science; and applications such as biostatistics, environmental

and social statistics; topology. This list is necessarily incomplete, and any research in which the

primary contribution is mathematical may be considered in this UOA, includin g experimental,

theoretical or computational investigations related to mathematical or statistical mode ls applied in

other subject areas.

82. The sub-panel expects submissions in this UOA from all areas of mathematical sciences, as

defined above, and expects that the majority of the research activity submitted will have made a

direct contribution to the UOA as characterised in the UOA descriptor. The sub-panel welcomes the

submission of innovative interdisciplinary research that incorporates mathematical, stati stical or

operational research content, irrespective of the primary research focus of the medium in which the

18

output is disseminated. It also expects to receive some outputs on the history of mathematical

sciences when they incorporate insights into mathematics or statistics.

83. The sub-panel will receive outputs describing purely pedagogic research within higher

education and where advice is required from Sub-panel 23 (Education), outputs w ill be cross-

referred. Operational research that is focused on business and management shou ld not normally be

submitted in this UOA.

Unit of Assessment 11: Computer Science and Informatics

84. The UOA includes the study and evaluation of methods for acquiring, storing, processing,

communicating and reasoning about information and interactivity in natural and engineered systems,

as characterised by the Association for Computing Machinery (ACM) Computing Classification

System, 2012 Revision https://www.acm.org/publications/class-2012. The sub-panel exp ects

submissions in this UOA to include contributions from ACM categories hardware, co mputer systems

organisation, networks, software and its engineering, theory of computation, mathematics o f

computing, information systems, security and privacy, human-centered computing, comput ing

methodologies, and applied computing. The field is characterised by strong theoretical f oundations

and systematic application of analysis, design, experimentation and evaluation.

85. The sub-panel expects that the majority of the research activ ity submitted will have made a

direct contribution to the UOA as characterised above, either by contributing new methods and

knowledge or through innovative applications of state-of-the-art methods to challenging problems in

other disciplines. Consequently, the sub-panel welcomes submissions containing interdisc iplinary

outputs that make contributions to other areas as well as computer science and informatics, though

outputs that apply routine computational methods may be better returned elsew here.

Unit of Assessment 12: Engineering

86. The UOA includes all areas of: aeronautical, mechanical and manufacturing engineering; bio-

engineering; chemical engineering; civil and construction engineering; electrical an d electronic

engineering; metallurgy, materials science and engineering; and general engineering. It includes, but

is not limited to, research carried out in: additive manufacturing; aerodynamics; aerospace

engineering; amorphous materials and glasses; antennae and radar; architectural engineer ing;

artificial intelligence and its applications; automotive engineering; avionics; bat tery technology;

biochemical engineering; bioengineering; biomaterials; biomedical engineering; building engineering

and physics; ceramics; climate change; combustion; communications and networks; co mposites;

computational methods; computer and software engineering; computer vision; construction and

infrastructure; construction management; construction materials; control and systems; corrosion

engineering; cryptography; data engineering; digital manufacturing; dynamics; earthquake

engineering; electrical power systems, machines and drives; electromagnetics and its applications;

electrochemical engineering; electronic devices; electronic materials; electronic systems and circuits;

energy and power engineering; energy harvesting and scavenging; engineering biology; engineering

design; engineering ethics; engineering management; environmental engineering; extre me events;

19

fire engineering; fluid mechanics; fluid power; fluid structure interactions; fluidics; food process

engineering; fuel technology; functional materials; geomatics and surveying; geosp atial engineering;

geotechnical engineering; health and safety; healthcare technologies; human factors and

ergonomics; hybrid materials; hydraulics and hydrology; information engineering; innovation

management and policy; instrumentation and measurement; intelligent and adaptiv e systems; life

cycle analysis; machine learning; manufacturing technology, processes and sy stems; maritime

engineering; materials engineering; materials for energy applications; materials character isation;

materials modelling; materials processing; mechanics; mechatronics; metals; metro logy;

microelectromechanical systems; mineral and mining engineering; modelling and simulation;

multimedia; music technology; nanoelectronics; nanomaterials; nanotechnology ; natural materials;

naval architecture; new materials; non-destructive testing and structural healt h assessment; nuclear

engineering; offshore and coastal engineering; particle technology; photonics and its applications;

polymer and large area electronics; polymers; power electronics; process and product engineering;

product design and verification; project management; quantum technologies; radio frequency

techniques; railway engineering; recycling and green technologies; renewable energy ; risk, reliability

and resilience; robotics and automation; sensors and actuators; signal and image process ing; solar

cells and systems; solid mechanics; space engineering; speech and language t echnology; structural

dynamics; structural materials; structures; surfaces and interfaces; sustainability engineering;

systems engineering; systems modelling and identification; technology and operation s management;

textiles; thermodynamics and heat transfer; tissue engineering; transportation engine ering; tribology

and wear; turbo-machinery and propulsion; vibration and acoustics; water and waste water

engineering; wind engineering; and wireless networks. It also includes pedagogic research in

engineering.

87. The sub-panel expects submissions in this UOA from all areas of engineering, as defined

above, and expects that the majority of the research activity submitted will have made a direct

contribution to the UOA as characterised in the UOA descriptor. It recognises and welcomes,

however, the increasingly interdisciplinary nature of research in this area, and expects that

submissions may contain outputs that not only make contributions to this UOA and other cognate

disciplines but also to UOAs that extend beyond traditional cognate discipline s.

Main Panel C: UOAs 13–24

Introduction

88. The sub-panels of Main Panel C cover a diverse range of content, discipl ines and

methodologies. The sub-panels anticipate receiving research outputs, impact case studies, and

impact and environment templates which reflect that rich diversity, and have no pre-conceptions

about where excellent research will be found.

89. Each sub-panel expects to receive submissions whose primary research focus falls within the

stated remit of its UOA. Submitting units are encouraged to submit their stron gest work, including

interdisciplinary work, in the UOA where it is most appropriate.

20

90. Criminology is a multi-disciplinary subject concerned with crime, criminals and criminal justice.

This could be submitted into various sub-panels. For the avoidance of doubt, it is recognised that

much criminological research may fall within the boundaries of Sub-panels 18 (Law), 20 (Social

Work and Social Policy) and 21 (Sociology). All three sub-panels welcome such wor k, which will be

assessed in accordance with the arrangements noted above, in particular making use of calibration

exercises, joint assessors and cross-referral as deemed appropriate by the sub-panels. Following

the assessment, the relevant sub-panels will review the health of UK criminolo gical research, and

will report on this in a discrete section of the panel overview reports.

Unit of assessment descriptors and boundaries

Unit of Assessment 13: Architecture, Built Environment and Planning

91. Descriptor: The UOA covers all forms of historical, t heoretical, technical, policy, applied and

practice-based research relevant to the planning, design, creation, functionality , use, conservation,

interpretation, assessment, management and governance of the built environment in both rural and

urban areas. This includes: architecture and related arts, building engineering, building surveying,

building sciences, climate change and disaster resilience, communities, construction, co nstruction

management, economic development, environment, health and well-being, housing, lan dscape,

manufacture, natural resources and ecosystem services, real estate, regenera tion, spatial analysis,

sustainability, transport, urban and regional planning and urbanism. It covers the social, economic,

legal, financial, environmental, technological, historic and cultural aspects of the built environment.

The UOA also covers any other research in which the built environment forms a major field for

application or provides the context for such research. It expects submissions in this UOA from a

broad range of disciplines, research methodologies and forms of output, across the spectrum of

fundamental, applied, pedagogical, policy and practice-based research. The submitted res earch may

span disciplinary and methodological boundaries.

92. Boundaries: The sub-panel expects submissions in this UOA from all areas outlined in the

UOA descriptor, but anticipates submissions that may span the boundaries between two or more

UOAs. Submitting units are encouraged to submit outputs that are of interdisciplinary nature, even if

the research is at the boundaries of the UOA.

Unit of Assessment 14: Geography and Environmental Studies

93. Descriptor: The UOA covers all aspects of research – conceptual, methodological,

substantive and applied – conducted within the disciplines of Geography and Environmental Studies,

as broadly defined, and contributing to their interdisciplinary collaborations. This research embraces

a wide range of enquiries into natural, environmental and human phenomena, and their

interrelationships in particular systems, contexts, periods and places, both in the UK and

internationally.

94. In Geography, submitted research may include work from all fields of physical and human

geography (for example, biogeography and ecosystem science; climatology; environmenta l

processes; environmental and climate change; forensic, geomorphology, glaciology , hydrology,

21

ocean and water science; Quaternary science; soil science; environmental geography, risks and

hazards; cultural, development, economic, health, heritage, historical, legal, political, population,

rural, social, transport, and urban geographies; geohumanities; and geographica l information

sciences and Earth observation); work that combines any of these fields (for example, in socio-

ecological systems and natural resource governance); and work that uses a wide range of available

methods, from science-based to humanistic and participatory, including numerical, t heoretical,

experimental, modelling, archival and field-based.

95. In Environmental Studies, submitted research may include work in any area of the field,

including those also present in environmental geography (for example, ecosys tem services and

natural capital, environmental economics, politics, policy and practice, sustainabl e development);

some aspects of environmental science (for example, conservation, ecology, environmental pollution

and resource management); and environmental assessment and decision support sy stems.

96. Boundaries: Given the breadth of the subject matter of UOA 14, it is inevitable that the

exemplification above is not exhaustive and that there will be some overlaps with other UOAs,

located both in Main Panel C and in other main panels and that submission may include work that is

close to the boundaries of the UOA. In areas where there is significant overlap betw een UOA 14 and

another UOA, it is expected that whole submissions will be made in the UOA appropriate to the

academic context and research environment in which the research was undertaken, an d with the

most appropriate range of expertise for the body of work as a whole.

Unit of Assessment 15: Archaeology

97. Descriptor: In Archaeology, submitted research may cov er all fields of the subject for

example, including but not limited to: archaeological theory and historiography; archaeological

science and archaeological methods; the archaeology of human origins and evolution, and

prehistoric and historic societies worldwide; early civilisations (including classical archaeology and

related historical and textual studies), medieval and post-medieval to contemporary archaeology;

colonial, industrial and maritime archaeology; landscape and environmental archaeology;

archaeological aspects of heritage; heritage science; museum studies; archaeological c onservation;

and forensic archaeology.

98. Boundaries: Submitting units are encouraged to submit their strongest work irrespective of

the form of output or the extent of its interdisciplinary nature, even if the resear ch is at the

boundaries of the UOA. There could be overlaps with any UOA, particularly UOA 1 (Clinical

Medicine), UOA 5 (Biological Sciences) in Main Panel A; UOA 7 (Earth Systems a nd Environmental

Sciences) in Main Panel B; UOA 14 (Geography and Environmental Studies) a nd UOA 22

(Anthropology and Development Studies) in Main Panel C; and UOA 28 (History) , UOA 29 (Classics)

and UOA 34 (Communication, Cultural and Media Studies, Library and Information M anagement) in

Main Panel D.

22

Unit of Assessment 16: Economics and Econometrics

99. Descriptor: The UOA is fully inclusive of all areas of economics and econometrics including

economic history. Research of all types – empirical or theoretical, applied, or policy-focused – will be

considered of equal standing.

100. Boundaries: Units are encouraged to submit their strongest w ork irrespective of the form of

output or the extent of its interdisciplinary nature, even if the research is at the boundaries of the

UOA. There could be overlaps with any UOA, including the other UOAs within Main Panel C,

particularly UOA 17 (Business and Management Studies). These parts of submission s will normally

be cross-referred, following advice from the cross-UOA members, to Sub-pa nel 17. In common with

any cross-referred work Sub-panel 16 (as the sub-panel for the UOA in w hich the work was

submitted for assessment) will retain responsibility for recommending the quality profi le.

Unit of Assessment 17: Business and Management Studies

101. Descriptor: The UOA includes (but is not restricted to) the areas of: accounting; banking;

business analytics; business and industrial economics; business ethics; business history; consumer

behaviour; corporate governance; corporate social responsibility; critical management studies;

employment relations; entrepreneurship; finance; human resource management; information

systems management; innovation management; international business; leader ship; management

education; management science; marketing; operations management; organisational psy chology;

organisational studies; project management; public policy; public sector management ; risk

management; service management; small firms; strategic management; supply chain management;

sustainability; technology management; third sector management; and any other field or sub-field

aligned to business and management.

102. Boundaries: Institutions are encouraged to submit their strongest work, irrespective of the

form of output. A variety of methodologies and interdisciplinary contributions is welcome. The sub-

panel anticipates that some of the work submitted in this UOA may overlap wit h the remits of UOA

10 (Mathematical Sciences), UOA 14 (Geography and Environmental Studies ) and UOA 16

(Economics and Econometrics).

103. The main panel’s preferred approach is that the majority of work submitted in a UOA is

assessed by that sub-panel. However, significant aspects of submissions in UOA 17 (Business and

Management Studies) are expected to fall within the remit of UOA 16 (Economics a nd

Econometrics). These parts of submissions, following advice from the cross-UOA member(s) will

normally be cross-referred to Sub-panel 16. In common with any cross-re ferred work, Sub-panel 17

(as the sub-panel for the UOA in which the work was submitted for assessment) w ill retain

responsibility for recommending the quality profile.

23

Unit of Assessment 18: Law

104. Descriptor: The UOA includes all doctrinal, theoretical, e mpirical, comparative, critical,

historical or other studies of law and legal phenomena including criminology, and socio-legal studies.

The sub-panel would also expect research on legal education to be submitted in this UOA.

105. Boundaries: All areas of law as described above fall within the boundaries of the UOA. Law is

a hybrid, multi-disciplinary subject which draws on disciplines in both the social sciences and the

humanities. Research in law may intersect with or draw upon a variety of disciplines and

methodologies. The sub-panel has been constituted with a broad spread of relevant expertise to

ensure informed assessment of all submissions, and encourages units to submit their strongest work

including research which is at the boundaries of the UOA.

106. For the avoidance of doubt, it is recognised that much criminological rese arch may fall within

the boundaries of Sub-panels 18 (Law), 20 (Social Work and Social Policy) and 21 (Sociology). All

three sub-panels welcome such work, which will be assessed in accordance with the arrangements

noted above, making use of calibration, joint assessors and cross-referral as deemed appropriate by

the sub-panels.

Unit of Assessment 19: Politics and International Studies

107. Descriptor: Politics and international studies has a broad spectrum, including: comparative

politics; area studies, international development, national and sub-national and grassroots politics;

studies of political institutions, public administration, policy and governance; the examination of

power, authority and legitimacy; political behaviour, political sociology and political economy; and

political theory and philosophy, including histories of political and international thought. It also

includes international relations theory; security studies including strategic, war and peace studies;

conflict research; international history; international political economy; and foreign pol icy analysis.

The sub-panel will welcome work from across this spectrum, including work that draws on a wide

range of theoretical approaches, among them feminist, postcolonial and queer perspectives, and on

diverse methods in political and international studies, among them quantitativ e, qualitative and

multimethod work as well as formal analysis and conceptual research. Outputs on pedagogic

research in politics and international studies will also be welcome.

108. Boundaries: The sub-panel expects to receive submission s in the UOA from all areas of the

discipline. It recognises that the boundaries of politics and international studies are not fixed and

welcomes work that is interdisciplinary, multidisciplinary and transdisciplinary in nature. The sub-

panel expects to read most of the material ‘in-house’. However, it will draw on the expertise of other

sub-panels where appropriate.

Unit of Assessment 20: Social Work and Social Policy

109. Descriptor: The UOA covers all forms of research i n social work, social policy and

administration, and criminology/criminal justice policy, gerontology, and substantive issues in these

studies. Research includes, but is not restricted to:

24

a. theory, methodology, empirical research, reviews/syntheses, analyses of documents,

records and statistics, ethics and values, and pedagogy as these areas apply to social

work, social care, social policy, criminology and criminal justice policy, geront ology and

substantive issues in these areas of study

b. research that defines and seeks to understand social problems and their impact

c. comparative research and research into international institutions and all forms of welfare

politics, policy and practice and conceptions of crime and criminal justice

d. research that uses a range of disciplinary and interdisciplinary approac hes including (but

not limited to): business and management, demography, development studies,

economics, education, geography, health studies, housing and urban studies, history ,

law, philosophy, politics, public policy, psychology, social anthropology and sociology

e. policy-making processes, practice, governance and management, serv ice design,

delivery and use, criminal justice design, and inter-professional relationships

f. research that includes links – and co-production – w ith, a variety of stakeholders (public

and private), professionals, volunteers, service users/participants and carers

g. research that analyses, evaluates and provides critical insights into the intersection of

the areas listed in a–e with key dimensions of representation and identity – particularly

marginal and/or excluded groups and communities – for example: social cla ss, gender,

sexuality, ‘race’/ethnicity, disability, age and migration/citizenship status.

110. Boundaries: Social work, social policy and administration, and criminology/criminal justice are

subjects closely related to a number of other disciplines within and outside the social sciences.

Political science, education, law, geography, social anthropology, developmental an d social

psychology, are clearly areas the boundaries of which overlap with this UOA – but certain types of

historical research and the ‘social aspects’ of, inter alia, environment studies, genetics/b iomedicine,

and engineering are also areas from which the sub-panel could expect to receive work.

111. Work submitted to this sub-panel may overlap significantly with the remit of Sub-panel 21

(Sociology). It is anticipated that the use of joint assessors and cross-referral of parts of submissions

may be required in order to ensure an appropriate assessment process, in ac cordance with the

arrangements in Part 5, paragraphs 399 to 404.

112. For the avoidance of doubt, it is recognised that much criminological rese arch may fall within

the boundaries of sub-panels, in particular Sub-panels 18 (Law), 20 (Social Work and Social Policy)

and 21 (Sociology). All three sub-panels welcome such work, which will be assessed in accordance

with the arrangements noted above, making use of calibration, joint assessors and cross-referral as

deemed appropriate by the sub-panels.

25

Unit of Assessment 21: Sociology

113. Descriptor: Sociology is a social science with a diversity of areas and approaches to the study

of social life and society. It includes empirical, critical and theoretical study of social structures,

power, cultures and everyday practices, including styles and material standards of living, opinions,

values and institutions. It includes analysis of and attention to social inequalities, divisions, justice

and solidarities at the micro, meso and macro levels. It covers all areas of social theory, historical

and comparative studies, and social research methodology, philosophy of social science, and

research on pedagogy in sociology. Sociology embraces a wide range of methodologies inclu ding

quantitative, qualitative and visual; and of all forms of data. The sub-panel also expects to consider

sociological research in such interdisciplinary fields as criminology and socio-legal studies, media

and cultural studies, social policy, gender and women’s studies, demography, socio -linguistics,

social psychology, psychosocial studies, social studies of science and technology, and lesbian, gay,

bisexual, transgender, queer and intersex studies.

114. The sub-panel expects submissions in this UOA from all fields of sociological enquiry

including, but not restricted to, research on cultures, economies and polities; class, race, ethnicity,

gender, sexuality, disability, and age, and their intersections; religion, education, hea lth and

medicine, family, media, welfare institutions, and work and employment; environ ment, technology

and the digital; and climate change; the body, interpersonal and inter-group relations, v iolence;

urban and rural issues; language and social interaction; political sociology, public policy and social

movements; political economy, globalisation, development, migration and diaspora; comparative

studies of societies of all kinds, including work on transnational structures and agencies, Europe

world systems.

115. As in previous research assessment exercises, work in inter disciplinary women’s and gender

studies may be submitted in this UOA, or may be cross-referred by other sub-panels to Sub-panel

21 (Sociology).

116. Work submitted in this UOA may overlap significantly with the remit of UOA 20 (Social Work

and Social Policy). It is anticipated that the use of calibration, joint assessors and cross-referra l of

parts of submissions may be required in order to ensure an appropriate assessment, in accordance

with the arrangements in Part 5, paragraphs 399 to 404.

117. For the avoidance of doubt, it is recognised that much criminological rese arch may fall within

the boundaries of Sub-panels 18 (Law), 20 (Social Work and Social Policy) and 21 (Sociology). All

three sub-panels welcome such work, which will be assessed in accordance with the arrangements

noted above, making use of calibration, joint assessors and cross-referral as deemed appropriate by

the sub-panels.

118. Boundaries: Sub-panel 21 (Sociology) acknowledge that the UOA does not specify

boundaries as per many other sub-panels. This is balanced through the descriptor outlin ing

processes for managing submissions, to a greater detail than other sub-panels. The sub-panel

consider issues of boundaries to have been addressed.

26

Unit of Assessment 22: Anthropology and Development Studies

119. Descriptor: The UOA covers all aspects of research within Anthropology and Development

Studies, including research that is conceptual, theoretical, empirical, applied, strategic and practice-

based, and that draws on a broad range of methodologies that includes the qualitativ e, quantitative,

field-based, ethnographic, laboratory-based, experimental, participatory, evaluative, visual and

comparative.

120. Anthropology is understood to include the broad fields of biological anthropology,

palaeoanthropology, and social and cultural anthropology. Social and cultural anthrop ology includes,

but is not limited to, economic and political anthropology; kinship, gender and relatednes s; religion;

cognition; medical anthropology; psychological anthropology; environment, conservation and

biodiversity; the anthropology of development; visual anthropology; ethnomusicology and

performance; material culture; and digital anthropology. Biological anthropology includes, but is not

limited to, human and non-human primate evolution and adaptation; palaeoanthropology; behaviour,

growth and development; health and disease; ecology; conservation; genetics; demograp hy; and for

forensic applications.

121. Development Studies involves the analysis of global, nat ional and local processes of change,

including social, economic, political, demographic, cultural, environmental and technological. These

may be studied from micro- to macro-scale and from local to global levels, with particular attention to

the relations between these. Attention is often paid to contexts characterised by poverty, inequalities,

environmental vulnerability and socio-political conflict and fragility. Research is often issue-driven

and involves critical interrogation of development and humanitarian theories, structures, processes,

policies and practices. Its focus can include, for example, poverty reduction, equa lities,

empowerment, peace and reduction of violence, and food security. It can be multidiscipl inary or

interdisciplinary and may combine social science with other disciplines.

Unit of Assessment 23: Education

122. Descriptor: All outputs in submissions under UOA 23 shou ld have an educational focus or

orientation. Research in education is closely related to a range of other disciplines w ith which it

shares common interests, methods and approaches. This diversity of content and methodology

requires the sub-panel to be flexible in setting out the boundaries of work relev ant to the REF.

123. The UOA may be broadly described as being concerned with research in the areas identified

in the following illustrative lists:

 Research which addresses education systems, issues, processes, provis ion and outcomes

in relation to sectors, such as: early years, primary, secondary, furt her, higher, medical,

workplace, adult and continuing education. It also includes teacher, healthcare and other

forms of professional education, vocational education and training; and informal, co mmunity

and lifelong learning.

27

 Research which addresses substantive areas, such as: curriculum, pedagogy , assessment,

language, teaching and learning; children, young people, student and adult learners; parents,

families and communities; culture, economy and society; teacher training, professi onalism

and continuing professional development (CPD); special and inclusive education;

participation, rights and equity issues; technology-enhanced learning; education policy; the

organisation, governance, management, effectiveness and improvement of educatio nal

institutions; education, training, workplaces, industry and the labour market; comparative,

international and development education.

 Research which employs a range of theoretical frameworks and methodologies drawn from

disciplinary traditions, including, but not limited to: anthropology, applied linguistics,

economics, geography, history, humanities, mathematics, statistics, philosophy , political

science, psychology, science and sociology. Research in the field of educatio n deploys a

range of qualitative and quantitative methodologies with structured, exploratory and

participatory research designs. These include, but are not limited to: surveys, experiment s

and controlled trials; ethnography, interview and narrative enquiry; action research an d case

study; evaluation research; critical theory and documentary analysis; analytic sy nthesis; and

systematic review.

124. The sub-panel accepts submissions in pedagogical research in higher education (whether or

not this is generated in education departments or similar units) and in professional educ ation

(including healthcare), while recognising that such work may instead be submitted in another

relevant UOA. The sub-panel will consider submissions in counselling and neurosc ience. However,

submissions in these areas may be referred to another sub-panel for advice.

Unit of Assessment 24: Sport and Exercise Sciences, Leisure and Tourism

125. Descriptor: Research in the UOA stems from the natural sciences, social sciences and the

humanities. The sub-panel expects to receive submissions from a wide range of disciplines and

subject areas that contribute to research in sport and exercise sciences, leisure and tourism. This

could include (in alphabetical order): adapted physical activity, anthropology, behaviour change,

biochemistry, biomechanics, business management and marketing, coaching, culture and media

studies, development studies, economics, education and pedagogy, engineering and technology,

festivals and events, geography, history, hospitality, law, medicine, molecular biology , motor learning

and control, nutrition, outdoor and adventure education, philosophy, physical education, physical

activity and health/public health, physiology, policy studies, politics, psychology , sociology, sports

injury and rehabilitation, and strength and conditioning. Research in sport and exercise sciences,

leisure and tourism is therefore derived from diverse disciplines and subject ar eas, and can also be

multidisciplinary and interdisciplinary.

126. Boundaries: The sub-panel expects submissions in this UOA of research of all types, and it

expects to consider research informed by a variety of research epistemologies, methodologies and

methods. The sub-panel will consider research defined as empirical, theoretical, strategic, applied,

or policy-focused as having equal standing.

28

Main Panel D: UOAs 25–34

Introduction

127. The main panel is charged with identifying excellence in the rich divers ity of research covered

by the UOAs described below. It welcomes all outputs arising from this res earch, in whatever genre,

medium or location, that can be demonstrated to meet the definition of research for the REF, as

outlined in Annex C of the ‘Guidance on submissions’ and that have entered the public domain

during the publication period. The sub-panels are committed to applying criteria and working

methods that reflect the distinctive character, methodologies and full breadth of these disciplines

(including interdisciplinary research), and that facilitate the formation of a balanced ran ge of

judgements, without privileging or disadvantaging any particular form of resea rch output, research

methodology or type of research environment.

128. The main panel and its sub-panels will operate according to the following princ iples:

 Panels will assess submissions in the form that HEIs hav e chosen to present their research,

within the REF framework.

 Panels will aim to identify excellence wherever they can find it.

Unit of assessment descriptors and boundaries

Unit of Assessment 25: Area Studies

129. Taking an inclusive view of Area Studies as a dynamic field, the sub-panel understands Area

Studies broadly to include the study of all regions of the world, across any period of time (ancient,

medieval and modern) however defined, and the communities associated with them. As well as in

terms of national territories, regions may be delineated in various ways, including by traditional

geographical designations (e.g. African Studies, American and Anglophone Studies (Ca nada and

the United States); Asian Studies including Central Asian, North East Asian S tudies (including China

and Japan), South Asian and South East Asian Studies; Latin American and Caribb ean Studies;

Australian, New Zealand and Pacific Studies; European Studies, including Russian and East

European Studies; Middle Eastern Studies (including Jewish and Islamic Studies)); by ecological

(e.g. Circumpolar Studies), geopolitical (e.g. Post-Soviet Studies) or institutional (e. g. European

Union Studies) criteria; or in terms of themes, processes or networks (e.g. Diaspora Stu dies, Post-

Colonial Studies, Gender Studies, Intercultural Studies etc.).

130. The sub-panel has expertise across the humanities and social sciences and welcomes work in

any language from any single-disciplinary, interdisciplinary, multidisciplinary, empiri cal or theoretical

perspective. It will assess submissions covering all, but not limited to, aspects of anthropology, law,

history, heritages, languages and linguistics, cultures, literatures, religions, philosophy, media,

society, economics, human geography, sociology, politics and international relations, and translation,

as well as inter-regional and globalisation studies. It welcomes ground-breaking or novel approaches

and seeks to reward innovation and excellence in both traditional and non-traditional form ats,

29

including applied, practice-based and pedagogical research. The sub-panel comprises sp ecialists in

humanities and social sciences and particularly welcomes work that crosses th e arts and

humanities/social sciences boundary.

131. The sub-panel has expertise to assess a wide range of work and takes an inclusive view of the

subject areas within its scope. Given the broad range of its descriptor, it recognises that submissions

may be made in this UOA that include work in languages, literatures, cultu res and societies falling

wholly or partially outside its members’ expertise. The sub-panel anticipates it will work closely with,

as appropriate, other sub-panels e.g. UOA 26 (Modern Languages and Linguistics) and UOA 19

(Politics and International Studies). The sub-panel is also mindful of the likely need to appoint further

panel members for the assessment stage of the REF.

132. The sub-panel recognises the overlapping boundaries in the sub-panel descri ptors, and that

aspects of research are naturally interdisciplinary or multi-disciplinary or span the boundaries

between individual UOAs. The sub-panel welcomes the submission of research t hat may equally be

submitted to other sub-panels. Where judged necessary by the panels, expert ise will be augmented

by additional assessors, or work will be cross-referred to relevant panels, according to the process

detailed in Part 5, paragraphs 399 to 404.

Unit of Assessment 26: Modern Languages and Linguistics

133. The UOA includes research on the languages, literatures, cultures and societies of all regions,

countries and communities where Celtic, Germanic, Romance or Slavonic languag es or other

languages of Europe and Latin America are, or were, used. This includes areas where European

languages have interacted with other cultures and languages. The UOA also includ es all areas of

general, historical, theoretical, descriptive and applied linguistics, phonetics, and translation and

interpreting studies, regardless of the methodology used or the language to which the studies are

applied. The sub-panel will take a broad view of what constitutes modern language stu dies. This will

include, but not be limited to: literature and thought; cultural studies; theatre studies; fil m and media

studies; visual cultures; language studies; translation and interpreting studies; polit ical, social and

historical studies; editorial scholarship, bibliography, textual criticism and theory and history of the

book; philosophy and critical theory; world literature and comparative literature; literature in relation

to the other arts; and applied, practice-based and pedagogical research, including translat ion and

creative writing. The sub-panel welcomes the submission of interdisciplinary research, including

work on language and literature in relation to science, medicine and technology , digital humanities,

or creative technologies, and will ensure that such work is assessed with appr opriate expertise.

134. The sub-panel has expertise to assess a wide range of work and takes an inclusive view of the

subject areas within its scope. Given the broad range of its descriptor, it recognises that submissions

may be made in this UOA that include work on languages, literatures, cultures and societies falling

wholly or partially outside its members’ expertise. The sub-panel consequently expects some degree

of overlap with UOA 25 (Area Studies). It anticipates that the two sub-panels will work together

closely and as appropriate before and during the assessment period.

30

135. The sub-panel recognises the overlapping boundaries in the sub-panel descri ptors, and that

aspects of research are naturally interdisciplinary or multidisciplinary or span the boundaries

between individual UOAs. The sub-panel welcomes the submission of research t hat may equally be

submitted to other sub-panels. Where judged necessary by the panels, expert ise will be augmented

by additional assessors, or work will be cross-referred to relevant panels, according to the process

detailed in Part 5, paragraphs 399 to 404.

Unit of Assessment 27: English Language and Literature

136. The UOA includes all aspects of language studies, including all areas of linguistics and of

applied linguistics, with primary reference to any variety of English or Scots; th e history of English or

Scots; Old Norse/Icelandic (language, literature and linguistic studies); English literature from the

early Middle Ages to the present day; North American literature; comparative lit erature; world

literatures in English; colonial and postcolonial literatures and languages; literatures translated into

English; women’s writing; creative writing and practice; life writing; children’ s literature; creative non-

fiction and/or creative critical writing; critical and cultural theory; cultural histor y; gender and sexuality

studies; editorial scholarship, bibliography, textual criticism and theory, and history of the book; Irish

literature in English; Scottish literature in English and Scots; Welsh literature in English; and applied,

practice-based and pedagogical research in English.

137. The sub-panel will take a broad view of what constitutes English literature and language. It

recognises the overlapping boundaries in the sub-panel descriptors, and that aspects of research

are naturally interdisciplinary or multi-disciplinary or span the boundaries between indiv idual UOAs.

The sub-panel aims to be inclusive, and welcomes the submission of research that may equally be

submitted to other sub-panels. This may include areas such as theatre and perfor mance studies;

cultural studies; film, television and digital media studies; popular music; history ; art history;

philosophy; the linguistics of languages other than those mentioned above; translation studies;

language and literature in relation to science, technology and medicine; medical and health

humanities; digital humanities; or creative technologies.

138. The sub-panel will ensure that such work is assessed with appropriate expertise. Where

judged necessary by the panels, expertise will be augmented by additional assessors, or work will

be cross-referred to relevant panels, according to the process detailed in Part 5, paragraphs 399 to

404. Specialist members of the relevant sub-panels will liaise to ensure that broad, cross-disciplinary

subject areas such as linguistics are appropriately assessed.

Unit of Assessment 28: History

139. The UOA includes all aspects of the study of the past.

140. The sub-panel expects submissions in this UOA from all areas of history. The following list (in

alphabetical order) is illustrative rather than exhaustive; it does not reflect any judgements about the

relative significance of the subject areas, nor does it specify ‘fields’: agricultural histor ies;

biographical histories; business histories; contemporary histories; cultural histories; ecclesiastical

histories; economic histories; histories of education; environmental histories; gender histori es; global

31

histories; heritage; historiography; history and memory; histories of Britain, Irel and and Continental

Europe (late Roman to the present); histories of ethnicity; histories of ideas; histories of North

America, South America, Africa, Asia and Australasia; histories of race; histories of science,

technology and medicine; histories of sexuality; histories of the book; indigenous histor ies;

imperial/colonial histories; international histories; labour histories; local and regional histor ies;

manuscript studies; material culture; media histories; military histories; oral histories; political

histories; public histories; religious histories; rural histories; social histories; theories of history;

transnational histories; urban histories; women’s histories; and world histori es.

141. All ancient history outputs will be automatically cross-referred to S ub-panel 29 (Classics).

Byzantine history outputs will also be cross-referred where it seems more appropriate for Sub-panel

29 to consider the output.

142. The sub-panel recognises the overlapping boundaries in the sub-panel descri ptors, and that

aspects of research are naturally interdisciplinary or multi-disciplinary or span the boundaries

between individual UOAs. The sub-panel welcomes the submission of research t hat may equally be

submitted to other sub-panels. Where judged necessary by the panels, expertise w ill be augmented

by additional assessors, or work will be cross-referred to relevant panels, according to the process

detailed in Part 5, paragraphs 399 to 404.

Unit of Assessment 29: Classics

143. The UOA includes the language, literature, history, culture, art, archaeology and thought

(including ancient science and philosophy) of Greece and Rome from the earliest times to late

antiquity; Latin language and literature of the Middle Ages and subsequent periods; Ancient Egypt

and the ancient Near East, Byzantine studies; modern Greek language, literature, history and

culture; the classical tradition; and the reception of these periods and subjects.

144. Within the boundaries are the following: the Greek world from the Bronze Age to the fall of the

Byzantine Empire; the Roman world from the Bronze Age to late antiquity; Gr eek lands, including the

Diaspora, from the medieval period to the present; the philology and linguistic s of Latin and Greek

and of related and neighbouring languages; comparative literature and such literature, literar y

theory, philosophy, political thought, material culture, art, film, television, digital media, creative

practices, music, and such political, archaeological and other cultural activity as exploits in any way

the history or cultural products of the Greek, Roman and Byzantine worlds, including translation and

performance of classical works; the pedagogy associated with learning and teaching in the subjects

listed here.

145. The list above is illustrative rather than exhaustive. It does not reflect any judgements about

the relative significance of the subject areas, nor does it specify ‘fields’.

146. The sub-panel recognises the overlapping boundaries in the sub-panel descri ptors, and that

aspects of research are naturally interdisciplinary or multi-disciplinary or span the boundaries

between individual UOAs. The sub-panel welcomes the submission of research t hat may equally be

32

submitted to other sub-panels. Where judged necessary by the panels, expertise w ill be augmented

by additional assessors, or work will be cross-referred to relevant panels, according to the process

detailed in Part 5, paragraphs 399 to 404.

Unit of Assessment 30: Philosophy

147. The UOA includes all areas and styles of, and approaches to, philoso phy. The sub‐panel

expects to receive submissions from all areas of philosophy, and considers the follow ing subjects

(listed alphabetically), among others, to be within the remit of the UOA: 19th to 21st century

European philosophy including phenomenology, existentialism, critical theory, hermene utics, and

deconstruction; aesthetics; applied philosophy; environmental philosophy; epistemology ; ethics,

including applied ethics and meta-ethics; feminist philosophy; history of philosophy including ancient,

medieval, modern and recent; logic; metaphysics; non-Western philosophy ; philosophy and history

of mathematics; philosophy and history of science, technology and medicine; phi losophy of

education; philosophy of language; philosophy of law; philosophy of mind; philosophy of race;

philosophy of religion; political and social philosophy. The areas mentioned are illustrative rather

than exhaustive, and do not reflect any judgement about the relative significance of the subject

areas.

148. Because philosophy engages with conceptual and foundational issues raised by other

disciplines, it spans boundaries with a number of other UOAs, including but not limited to all the

other UOAs within Main Panel D and the following UOAs within other main panels: UOA 1 (Clinical

Medicine), for example philosophy of biological sciences; UOA 2 (Public Health, Health Services and

Primary Care), for example medical ethics; UOA 4 (Psychology, Psychiatry and Neuroscience), for

example cognitive science; UOA 5 (Biological Sciences), for example philosop hy of biological

sciences; UOA 9 (Physics), for example the philosophy of physics; UOA 10 (Mathematica l

Sciences), for example mathematical logic; UOA 16 (Economics and Econom etrics), for example

social choice theory and game theory; UOA 18 (Law), for example jurisprudence; UOA 19 (Politics

and International Studies), for example political theory; UOA 21 (Sociology), for example social

theory.

149. In accord with the Philosophy sub-panel’s aim to be inclus ive, it covers all types of applied

philosophy relating to practical issues both within and outside academia. The remit also covers work

concerned with philosophical questions raised by other disciplines, for example work concerned with

the foundations, methods, epistemic status, or interpretation of findings or theories in the other

disciplines. The sub-panel may consider that work that merely references philosoph ical ideas without

engaging with them philosophically will have its excellence best assessed by another sub-panel, and

will consider cross-referral accordingly.

150. The sub-panel recognises the overlapping boundaries in the sub-panel descri ptors, and that

aspects of research are naturally interdisciplinary or multi-disciplinary or span the boundaries

between individual UOAs. The sub-panel welcomes the submission of research t hat may equally be

submitted to other sub-panels. Where judged necessary by the panels, expert ise will be augmented

 
33 
 
 
by additional assessors, or work will be cross-referred to relevant panels,                        according to the process 
detailed in Part 5, paragraphs 399 to 404. 
Unit of Assessment 31: Theology and Religious Studies 
151.  The UOA encompasses all research in theology and religion,
                        and is inclusive of all disciplinary 
                    
approaches adopted in the field, including philosophical, theological, historical,
                        philolog
                        ical, literary, 
phenomenological, psychological, sociological and anthropological methodologies.
                        It is also inclusive 
of research into the nature of theology and/or religious studies as disciplines,                        and of the 
methodologies they employ. 
152.  It encompasses the study and interpretation of religious institutions,
                        movements, texts, laws, 
                    
practices, ethics, beliefs, symbols, media, social relations, material objects,
                        sp
                        aces and flows, both 
historical and contemporary in local and/or global contexts. It includes all religious
                        tradit
                        ions, 
spiritualities and sacralised forms of commitment and their expression in different
                        cultural media – 
for example film, art, music and literature, in whatever genre or medium. The
                        study of varieties of 
secularism, secularity and non-belief which reference religion explicitly or implicitly
                        is also included. It 
also covers work concerned with theological and religious questions raised
                        by other disciplines, 
including being able to review work relating to the context, assumptions and content
                        of religious 
education in different settings. 
153.  Theology and Religious Studies is an inherently multi- and cross-disc
                        iplinary subject, and 
                    
religion intersects with many other aspects of society, politics and culture. In
                        recognition of this, the 
sub-panel will welcome submissions which overlap with the remit of other UOAs;
                        or for which UOA 
31 is not the only appropriate one; or from those undertaking relevant research
                        in academic units not 
classified as theology, divinity or religious studies; or from academic units which
                        specia
                        lise in only 
one area of the field. 
154.  Given the multi-disciplinary reach of UOA 31, it is anticipated that
                        a substantial portion of 
                    
submissions received will overlap with other UOAs, for example with UOA 14
                        (Geography
                         and 
Environmental Studies), UOA 18 (Law), UOA 19 (Politics and International Stu
                        dies), UOA 21 
(Sociology), UOA 22 (Anthropology and Development Studies), UOA 23
                        (Edu
                        cation) and the UOAs 
                        
                    
within Main Panel D. Sub-panel 31 contains considerable linguistic, methodological
                        and cross-
disciplinary expertise, but will apply the arrangements set out in Part 5, paragraphs
                        399 to 404, 
                    
where expertise needs to be augmented. Sub-panel 31 continues to welcome innovat
                        ive and cross-
disciplinary approaches to the study of religion as well as more traditional methods.
                    
155.  The sub-panel recognises the overlapping boundaries in the sub-panel
                        descri
                        ptors, and that 
                    
aspects of research are naturally interdisciplinary or multi-disciplinary or span
                        the boundaries 
between individual UOAs. The sub-panel welcomes the submission of research
                        that
                         may equally be 
submitted to other sub-panels. Where judged necessary by the panels, expertise
                        w
                        ill be augmented 
by additional assessors, or work will be cross-referred to relevant panels,                        according to the process 
detailed in Part 5, paragraphs 399 to 404. 

34

Unit of Assessment 32: Art and Design: History, Practice and Theory

156. The sub-panel will assess research from all aspects of the history, t heory and practice of art

and design, and disciplines where these relate to visual, material and spatial cultures. The sub-panel

will consider outputs, in whatever genre or medium, that meet the definition of research (as outlined

in ‘Guidance on submissions’, Annex C). The sub-panel acknowledges and welcomes a diverse

range of methods and approaches to research, and therefore adopts an inclusive definiti on of its

remit.

157. It is anticipated that outputs will span a range of texts, edited p ublications and creative

practices, as well as artefacts, events and curatorial outputs. The sub-panel expect s to evaluate

research that encompasses analytical, applied, ethnographical, experimental, historical,

pedagogical, scientific, technological and theoretical approaches to the widest domains of the

history, theory and practice of art and design, and covers the broadest understanding o f the subject

disciplines within any cultural, geographical or historical context.

158. The sub-panel is committed to applying criteria and working methods that are appropriate to all

submitting units, whatever their size or structure, without privileging any particular form of research

output or environment.

159. The sub-panel recognises the overlapping boundaries in the sub-panel descri ptors, and that

aspects of research are naturally interdisciplinary or multi-disciplinary or span the boundaries

between individual UOAs. The sub-panel welcomes the submission of research t hat may equally be

submitted to other sub-panels. Where judged necessary by the panels, expert ise will be augmented

by additional assessors, or work will be cross-referred to relevant panels, according to the process

detailed in Part 5, paragraphs 399 to 404.

Unit of Assessment 33: Music, Drama, Dance, Performing Arts, Film and Screen

Studies

160. The sub-panel will assess research from all areas of music, drama, dance, theatre,

performance, live and sonic art, film, television and screen studies. It anticipates that outputs will

span a range of artefacts, creative practices, curatorial outputs, edited publications, record ings and

writings. The sub-panel expects to evaluate research that encompasses analytical, ap plied, critical,

ethnographical, historical, interdisciplinary, pedagogical, practice-based, scientific, technologi cal and

theoretical approaches to all of the subject areas indicated above. It covers the broadest

understanding of the subject disciplines within, between and across any cultural, geograp hical or

historical contexts.

161. The sub-panel recognises the overlapping boundaries in the sub-panel descri ptors, and that

aspects of research are naturally interdisciplinary or multi-disciplinary or span the boundaries

between individual UOAs. The sub-panel welcomes the submission of research t hat may equally be

submitted to other sub-panels. Where judged necessary by the panels, expert ise will be augmented

by additional assessors, or work will be cross-referred to relevant panels, according to the process

detailed in Part 5, paragraphs 399 to 404.

35

Unit of Assessment 34: Communication, Cultural and Media Studies, Library and

Information Management

162. The sub-panel recognises the rich diversity of research across the Arts, Humanities and Social

Sciences in communication, cultural and media studies, library and information management, and

welcomes all outputs arising from this research, in whatever genre or medium, that can be

demonstrated to meet the definition of research for the REF (as outlined in ‘Guidance on

submissions’, Annex C). In setting out its remit, the sub-panel recognises that the UOA descriptor

covers two broad fields of research which are often distinct both organisationally and academically,

and welcomes submissions that reflect this. It also recognises that the activitie s covered by its remit,

even within its two broad fields of coverage, are often rooted in quite distinct research traditions or

infrastructures. It will assess research on its merits, with no penalty for research which is plainly

within a distinct tradition within the sub-panel’s remit. It will nonethele ss welcome research which

seeks to engage with questions and concerns, such as the ‘information society ’, heritage (both

cultural and museum aspects), networks or convergence, which may transcend field boundaries.

163. The UOA includes research that addresses or deploys theory, history, institutional, policy,

textual, critical and/or empirical analysis, or practice within communication, culture, media,

journalism, film, television and screen studies. Within UK higher education much, but not all, of this

work is likely to emanate from units or departments in communication studies, cultura l studies,

media studies, journalism, or film and screen/television studies. This work will include research on

online and screen-based media (such as film, television, games and other digital for ms), print media,

computer-mediated communication, digital infrastructure and platform studies focuse d on data and

society, diverse information and communication technologies, cultural policy, the creative industries

and popular culture, which will be variably titled and organised. The sub -panel will assess research

as defined above which addresses (but is not confined to): policy for regulation of culture and the

media and communication industries; the organisation, institutions, political eco nomy and practice of

cultural production; media and cultural texts, forms and practices; media and cultural audiences,

consumption and reception; the role of changing technology, including emergent digit al technologies,

in media production, content manipulation, distribution, access and participation. It is recognised that

this will include work which explores questions of power, identity and difference in relation to media,

communication and cultural studies which may sit at the intersections of (among others) gender and

sexuality studies, race and postcolonial studies, and disability studies.

164. The UOA also includes research concerned with the management of information and

knowledge in all formats, namely librarianship and information science, archiv es and records

management, and information systems. This concerns research on the generat ion, organisation,

dissemination and publication, exploitation, protection, and evaluation of informat ion and knowledge,

and the impacts of such activities. It may include, for example, research that focuses on digital

humanities; digital participation; information behaviour and use; information ethics; information

literacy; information media; information policy; information retrieval; informat ion security; information

seeking; the information society; knowledge management systems; preservation and conservation;

systems thinking; systems development; and the cultural, economic, ethical, historica l, philosophical,

and societal aspects of the disciplines and their associated professions.

36

165. The sub-panel recognises the overlapping boundaries in the sub-panel descri ptors, and that

aspects of research are naturally interdisciplinary or multi-disciplinary or span the boundaries

between individual UOAs. The sub-panel welcomes the submission of research t hat may equally be

submitted to other sub-panels. Where judged necessary by the panels, expert ise will be augmented

by additional assessors, or work will be cross-referred to relevant panels, according to the process

detailed in Part 5, paragraphs 399 to 404.

 
37 
 
 
Part 3: Assessment criteria 
Section 1: Submissions 
Interdisciplinary research 
166.  The REF main and sub-panels welcome the submission of interdisciplinary
                         research, as 
                    
defined in paragraph 390, in any relevant UOA.  
167.  The arrangements for assessing interdisciplinary research –                        including through the guidance 
                    
provided by the interdisciplinary advisers on the main and sub-panels, the interdisciplinary
                         research 
identifier, and the appointment of additional members and assessors – are se
                        t out in the working 
methods (see paragraphs 390 to 398). 
                        
                    
168.  The interdisciplinary research identifier for outputs allows HEIs
                        to identify in the submission 
                    
those outputs which it considers to be interdisciplinary, and to draw
                        this to the panels’ attention. This 
                    
is intended to give greater confidence to HEIs to submit interdisciplinary
                        resear
                        ch, and will allow 
panels, working with their IDR advisers, to consider the most appropriate means
                        of assessing the 
output. It is recognised that submissions may cover a broad range of disciplines,
                        or may not have an 
identifiable disciplinary focus. The sub-panels therefore encourage HEIs to identify
                        in submissions all 
outputs they consider meet the definition of interdisciplinary research (set out                        in paragraph 390) 
where they wish to draw this to the panels’ attention. The sub-panels recogn
                        ise that outputs flagged 
                    
as interdisciplinary may incorporate research crossing main panel areas, may
                        span disciplines 
across the sub-panels within a main panel, or may incorporate research areas
                        covered within a sub-
panel – particularly where UOAs cover a broad range of disciplines. 
169.  There will be no advantage or disadvantage in the assessment                        in identifying outputs as 
                    
interdisciplinary. The main and sub-panels will apply the standards of excellen
                        ce defined by the 
starred quality levels equally to research in interdisciplinary areas and to research
                        within distinct 
disciplines. The main and sub-panels consider that all such research
                        is capable of displaying the 
                    
highest standards of quality. 
Work on the boundaries between UOAs 
170.  The REF main and sub-panels recognise the diverse nature of the
                        disciplines that they cover, 
                    
that UOAs do not have firm or rigidly definable boundaries, and that aspects of
                        research span the 
boundaries between individual UOAs, whether within a main panel or across main
                        p
                        anels. They also 
recognise that there are research areas which may be undertaken in a range
                        of different contexts, 
and some of these therefore occur in the descriptors of a number of UOAs. The
                        main and sub-
panels welcome the submission of such research, in any relevant UOA. 
171.  The arrangements for assessing submissions that span UOA boundaries
                        – including the 
                    
cross-referral process – are set out in the working methods (see paragraph
                        s 399 to 404). The UOA 
                        
                    
descriptors indicate where the panels might expect work submitted in their UOA
                        to cross boundaries 
with other UOAs, but recognise that there may be other overlaps. 

38

172. Panels will assess, on an equal basis, submissions that reflect the work of administrative units

such as departments, and submissions that do not map neatly onto departmental or other

administrative structures within HEIs. In either case, institutions will not be penalised if submissions

contain some work that overlaps UOA boundaries. The main and sub-panels will apply the standards

of excellence defined by the starred quality levels equally to research that spans UOA boundaries

and to research solely within the remit of one UOA. The main and sub-panels cons ider that all such

research is capable of displaying the highest standards of quality.

Pedagogic research

Main Panel A criteria – pedagogic research

173. It is expected that research on pedagogy or medical or veterinary education will be

submitted in UOA 23 (Education) and research on medical ethics will be submitt ed in UOA 30

(Philosophy). Research on the philosophical and ethical aspects of healthcare and on education

relevant to its disciplines may be submitted in UOA 3 (Allied Health Professions, Dentistry,

Nursing and Pharmacy) and any such outputs are expected to be assessed by that panel.

Main Panel B criteria – pedagogic research

174. Research on pedagogy and educational issues within higher education that relate to the

disciplines covered by Main Panel B may be submitted in the UOA to which it relates rather than

to UOA 23 (Education). Such research will be assessed by the sub-panel for the UOA in which it

is submitted and where advice is required from Sub-panel 23 (Education), outputs will be cross-

referred.

175. Bodies of research into teaching in other education sect ors or on general educational

issues should be submitted in UOA 23 (Education). Individual outputs on these issues received by

the sub-panels in Main Panel B will normally be cross-referred to Sub-panel 23 (Educat ion) as

appropriate.

Main Panel C criteria – pedagogic research

176. Research on pedagogy and edu cational issues within higher e ducation that relate to the

disciplines covered by Main Panel C may be submitted in the UOA to wh ich it relates or in UOA 23

(Education), as deemed appropriate by submitting HEIs.

Main Panel D criteria – pedagogic research

39

177. Research on pedagogy and educational issues that relate to the discipline s covered by

Main Panel D may be submitted in the UOA to which it relates or in UOA 23 (Educatio n), as

deemed appropriate by submitting HEIs. Main Panel D anticipates that individ ual sub-panels will

normally assess such research where it relates to education in the sub-panel’s discip line area.

Multiple submissions

178. ‘Guidance on submissions’ (paragraphs 73 to 77) sets out the arrangements whereby

institutions may exceptionally, and only with prior permission from the REF director, make more than

one submission (multiple submissions) in the same UOA. These exceptions are:

a. Where an institution involved in a joint submission wishes to make an additional

individual submission in the same UOA.

b. Multiple submissions to Sub-panel 26 (Modern Languages and Linguistics) w ill be

permitted where one submission is in Celtic Studies and the other in Modern Lang uages

and Linguistics. This has been agreed in recognition of the special cultural significance

of Celtic Studies in parts of the UK, and the particular legal status of the Welsh languag e

in Wales. For other types of multiple submissions in UOA 26, see paragraph 185.

c. Where HEIs merged after 1 July 2018, they may seek permission to ma ke two separate

submissions in all of the UOAs in which they wish to submit, if, for example, they

anticipate difficulty in achieving academic cohesion between the merger date and the

submission date. Permission is unlikely to be granted to such HEIs to make se parate

submissions only in some of the UOAs in which they wish to submit. In the ev ent that

HEIs merged prior to 1 July 2018, the merged HEI should normally make one

submission only to each UOA.

d. Where a sub-panel considers there is a case for multiple submissions in its UOA, given

the nature of the disciplines covered, the institution may request a multiple submissio n.

The panels’ expectations are set out in paragraphs 180 to 186.

179. Each submission will be awarded a single overall quality profile. Where a single submission

includes distinct organisational units or areas of research and where the REF sub-panel considers it

appropriate, the sub-panel will provide feedback to the head of institution relating to the distinct units

or areas of research.

Main Panel A supplementary criteria – multiple submissions

40

180. Sub-panels in Main Panel A do not consider that there is a case for multiple submissions in

their UOAs, based on the nature of the disciplines covered, and do not expect to receive requests

for multiple submissions in these UOAs (other than for the reasons stated in paragraph 178 a.-c.).

181. Sub-panel 3 (Allied Health Professions, Dentistry, Nursing and Pharm acy) recognises that

institutions may wish to receive output sub-profiles for distinct areas covered in their submission

and will provide output sub-profiles against the following areas to the head of institution where

requested: Nursing, Dentistry, Pharmacy, Allied Health Professions and Biom edical Sciences. If

institutions wish to receive output sub-profiles, they should assign each output in their submission

to one of the above categories. Sub-profiles will not be provided for outputs which have not been

tagged. The normal expectation is that output sub-profiles will not be provided where t here is a

small number of staff associated with those outputs (typically less than five FTE). Output sub-

profiles will be provided confidentially to the head of the institution concerned

Main Panel B supplementary criteria – multiple submissions

182. Sub-panels in Main Panel B do not consider that there is a case for multiple submissions in

their UOAs, based on the nature of the disciplines covered, and do not expect t o receive requests

for multiple submissions in these UOAs (other than for the reasons stated in paragraph 178 a.-c.).

183. Sub-panel 12 (Engineering) recognises that institutions may wish to receive output sub-

profiles for distinct areas of engineering covered in their submission and will provide output sub-

profiles against the following areas to the head of institution where requested: Aeronautical,

Mechanical and Manufacturing Engineering; Bio-engineering; Civil and Construction Eng ineering;

Chemical Engineering; Electrical and Electronic Engineering; General Engineering; Metallurgy

and Materials. If institutions wish to receive output sub-profiles, they should assign e ach output in

their submission to one of the above categories. Sub-profiles will not be provided for outputs

which have not been tagged. If a ‘combined’ sub-profile is required for all remaining outputs (as

different from the overall output profile) then institutions should assign all remaining outputs to

general engineering. The normal expectation is that output sub-profiles will not be provided where

there is a small number of staff associated with those outputs (typically less than five FTE).

Output sub-profiles will be provided confidentially to the head of the institution concerned.

Main Panel C supplementary criteria – multiple submissions

184. In Sub-panel 22 (Anthropology and Development Studies) requests for multiple submissions

where institutions have separate Anthropology and Development Studies (or Internatio nal

Development) departments would normally fulfil the criteria. The remainder of s ub-panels in Main

Panel C do not consider that there is a case for multiple submissions in their UOAs, based on the

41

nature of the disciplines covered, and do not expect to receive requests for multip le submissions

(other than for the reasons stated in paragraph 178 a.-c.).

Main Panel D supplementary criteria – multiple submissions

185. The following sub-panels in Main Panel D consider that there is a case, based on the nature

of the disciplines covered by their UOAs, for multiple submissions in these UO As and would

expect to receive requests for:

 Sub-panel 25 (Area Studies)

 Sub-panel 26 (Modern Languages and Linguistics)

 Sub-panel 32 (Art and Design: History, Practice and Theory )

 Sub-panel 33 (Music, Drama, Dance, Performing Arts, Film and Screen Studies)

 Sub-panel 34 (Communication, Cultural and Media Studies, Library and Information

Management)

186. Requests for multiple submissions may be made in other UOAs within Main Panel D but are

expected to be a rare occurrence. All such requests will be considered according to the criteria

and procedures in paragraphs 73 to 77 of ‘Guidance on submissions’.

Section 2: Staff

Independent researchers

187. ‘Guidance on submissions’ (paragraph 128) sets out that staff employed on ‘research only’

contracts must be independent researchers to meet the definition of Category A eligible. For the

purposes of the REF, an independent researcher is defined as an individual who undert akes self-

directed research, rather than carrying out another individual’s research programme.

188. Possible indicators of independence are listed below. Institutions should note that each

indicator may not individually demonstrate independence and, where appro priate, multiple factors

may need to be considered. Across all main panels, the following indicators wo uld normally identify

research independence:

 leading or acting as principal investigator or equivalent on an externally funded research

project

42

 holding an independently won, competitively awarded fellowsh ip where research

independence is a requirement. An illustrative, but not exhaustive, list of independe nt

fellowships can be found at www.ref.ac.uk, under Guidance

 leading a research group or a substantial or specialised wor k package.

Main Panels C and D supplementary criteria – independent researchers

189. In addition to the generic criteria specified in the ‘Guidance o n submissions’, Main Panels C

and D also consider that the following attributes may generally indicate research ind ependence in

their disciplines:

• Being named as a Co-I on an externally funded research grant/award.

• Having significant input into the design, conduct and interpretation of the research.

Section 3: Outputs

Criteria and level definitions

190. This section provides a descriptive account of how the sub-panels will interpret and apply the

generic criteria for assessing outputs and the starred quality levels. This descript ive account

expands on and complements the generic criteria and definitions in Annex A of ‘G uidance on

submissions’, but does not replace them.

191. Originality will be understood as the extent to which the output makes an important and

innovative contribution to understanding and knowledge in the field. Research outputs that

demonstrate originality may do one or more of the following: produce and interpret new empirical

findings or new material; engage with new and/or complex problems; develop innov ative research

methods, methodologies and analytical techniques; show imaginative and creative scope; provide

new arguments and/or new forms of expression, formal innovations, interpretat ions and/or insights;

collect and engage with novel types of data; and/or advance theory or the analysis of doctrine, policy

or practice, and new forms of expression.

192. Significance will be understood as the extent to which the work has influenced, or has the

capacity to influence, knowledge and scholarly thought, or the development and understanding of

policy and/or practice.

193. Rigour will be understood as the extent to which the work demonstrates intellectual coherence

and integrity, and adopts robust and appropriate concepts, analyses, sources, theories and/or

methodologies.

43

194. The generic definitions of the starred quality levels in the overall quality profile in each of the

three sub-profiles – outputs, impact and environment – are in Annex A of ‘Guidance on

submissions’. The panels would like to emphasise that ‘world-leading’, ‘intern ationally’ and

‘nationally’ in this context refer to quality standards. They do not refer to the nature or geographical

scope of particular subjects, nor to the locus of research, nor its place of dissemination.

195. The main panels have set out below a descriptive account of the starred level definitions for

outputs, as they apply in each main panel. These are provided to inform their subject communities

about how the panels will apply the definitions in making their judgements. Variations in terminology

reflect disciplinary norms but do not indicate a difference in the quality standards themselves. These

descriptive accounts should be read alongside, but do not replace, the generic defin itions.

Interdisciplinary research

196. Interdisciplinary outputs will be assessed against the generic criteria of originality, significance

and rigour. In assessing interdisciplinary outputs, the sub-panels will make use of guidance provided

by the Interdisciplinary Research Advisory Panel (IDAP) that originality and significance can be

identified in one, some or all of the constituent parts brought together in the work, or in their

integration; they do not need to be demonstrated across all contributing areas/fields . This guidance

will work in parallel with – rather than replace – the generic criter ia of originality, significance and

rigour.

Main Panel A supplementary criteria – level definitions

197. In assessing outputs, the sub-panels will look for evidence of the quality of the output in

terms of its originality, significance and rigour, and will apply the generic definitions of the starred

quality levels.

198. The sub-panels will look for evidence of some of the following ty pes of characteristics of

quality, as appropriate to each of the starred quality levels:

 scientific rigour and excellence, with regard to design, method, execution and analysis

 significant addition to knowledge and to the conceptual framework of the field

 actual significance of the research

 the scale, challenge and logistical difficulty posed by the research

 the logical coherence of argument

 contribution to theory-building

 significance of work to advance knowledge, skills, understanding and scholar ship in

theory, practice, education, management and/or policy

44

 applicability and significance to the relevant service users and research users

 potential applicability for policy in, for example, health, healthcare, public health, food

security, animal health or welfare.

199. Unless there is sufficient evidence of at least one of the above, or the definition of research

used for the REF is not met, research outputs will be graded as ‘unclassified’.

200. The sub-panels welcome research practice that supports reproducible scienc e and the

application of best practice. Examples include registered reports, pre-registration, pub lication of

data sets, experimental materials, analytic code, and use of reporting checklists for publication

purposes and those relating to the use of animals in research. These contribute to the evaluation

of rigour for submitted outputs. Replication studies may be submitted as outputs and will be

evaluated on the extent to which they contribute significant new knowledge, improv ed methods, or

advance theory or practice

1

.

201. The sub-panels will use citation information, where appropriate and available, as part of the

indication of academic significance to inform their assessment of output quality. Furt her details on

the use of citation data are provided in paragraphs 274 to 276.

Main Panel B supplementary criteria – level definitions

202. In assessing outputs, the sub-panels will look for evidence of originality, significance and

rigour and apply the generic definitions of the starred quality levels as follows:

a. In assessing work as being four star (quality that is world-leading in terms of

originality, significance and rigour), sub-panels will expect to see evidence of, or

potential for, some of the following types of characteristics:

 agenda-setting

 research that is leading or at the forefront of the research area

 great novelty in developing new thinking, new techniques or novel results

 major influence on a research theme or field

1

Institutions may find it useful to refer to international guidelines such as the following:

ARRIVE https://www.nc3rs.org.uk/arrive-guidelines

CONSORT http://www.consort-statement.org/

PRISMA http://www.prisma-statement.org/

COPE http://publicationethics.org/

ICMJE http://www.icmje.org/

ITHENTICATE http://www.ithenticate.com/

45

 developing new paradigms or fundamental new concepts for research

 major changes in policy or practice

 major influence on processes, production and management

 major influence on user engagement.

b. In assessing work as being three star (quality that is internationally excellent in terms

of originality, significance and rigour but which falls short of the highest standards of

excellence), sub-panels will expect to see evidence of, or potential for, some of the

following types of characteristics:

 makes important contributions to the field at an international standar d

 contributes important knowledge, ideas and techniques which are likely to have a

lasting influence, but are not necessarily leading to fundamental new concepts

 significant changes to policies or practices

 significant influence on processes, production and management

 significant influence on user engagement.

c. In assessing work as being two star (quality that is recognised internationally in

terms of originality, significance and rigour), sub-panels will expect to see evidence

of, or potential for, some of the following types of characteristics:

 provides useful knowledge and influences the field

 involves incremental advances, which might include new knowledg e which

conforms with existing ideas and paradigms, or model calculations using

established techniques or approaches

 influence on policy or practice

 influence on processes, production and management

 influence on user engagement.

d. In assessing work as being one star (quality that is recognised nationally in terms of

originality, significance and rigour), sub-panels will expect to see evidence of, or

potential for, some of the following types of characteristics:

 useful but unlikely to have more than a minor influence in the field

46

 minor influence on policy or practice

 minor influence on processes, production and management

 minor influence on user engagement.

e. Research will be graded as ‘unclassified’ if it falls below the quality levels described

above or does not meet the definition of research used for the REF.

Main Panel C supplementary criteria – level definitions

203. In assessing outputs, the su b-panels will look for evidence o f originality, significance and

rigour, and apply the generic definit ions of the starred quality levels as follows:

a. In assessing work as bei ng four star (quality that is world-leading in terms of

originality, significance and rigour), sub-panels will expect to see so me of the following

characteristics:

 outstandingly novel in deve loping concepts, paradigms, techniques or outcomes

 a primary or essential point of reference

 a formative influence on t he intellectual agenda

 application of exceptiona lly rigorous research desig n and techniques of

investigation and analysis

 generation of an exceptiona lly significant data set or resear ch resource.

b. In assessing work as being three star (quality that is internationally excellent in terms

of originality, significance and rigour but which falls short of the highest standards of

excellence), sub-panels will expect to see some of the following characteristics:

 novel in developing conc epts, paradigms, technique s or outcomes

 an important point of refer ence

 contributing very important knowledge, ideas and techniques whic h are likely to

have a lasting influence on the intellectual agenda

 application of robust and appr opriate research design and techni ques of

investigation and analysis

47

 generation of a substantial data set or research resource.

c. In assessing work as being two star (quality that is recognised internationally in

terms of originality, significance and rigour), sub-panels will expect to see some of the

following characteristics:

 providing important knowledge and the application of such know ledge

 contributing to incremental and cumulative advances in knowledge

 thorough and professional application of appropriate research design and

techniques of investigation and analysis.

d. In assessing work as being one star (quality that is recognised nationally in terms of

originality, significance and rigour), sub-panels will expect to see some of the

following characteristics:

 providing useful knowledge, but unlikely to have more than a minor influence

 an identifiable contribution to understanding, but largely framed by existing

paradigms or traditions of enquiry

 competent application of appropriate research design and techniques of

investigation and analysis.

e. Research will be graded as ‘unclassified’ if it falls below the quality levels described

above or does not meet the definition of research used for the REF.

Main Panel D supplementary criteria – level definitions

Interpretation of generic level definitions

204. The terms ‘world-leading’, ‘international’ and ‘natio nal’ will be taken as quality benchmarks

within the generic definitions of the quality levels. They will relate to the actual, likely or deserved

influence of the work, whether in the UK, a particular country or region outside the UK, or on

international audiences more broadly. There will be no assumption of any necessary international

exposure in terms of publication or reception, or any necessary research content in terms of topic

or approach. Nor will there be an assumption that work published in a language other than

English or Welsh is necessarily of a quality that is or is not intern ationally benchmarked.

205. In assessing outputs, the sub-panels will look for evidence of originality, significance and

rigour and apply the generic definitions of the starred quality levels as follows:

48

a. In assessing work as being four star (quality that is world-leading in terms of

originality, significance and rigour), sub-panels will expect to see evidence of, or

potential for, some of the following types of characteristics across and possibly

beyond its area/field:

 a primary or essential point of reference

 of profound influence

 instrumental in developing new thinking, practices, paradigms, policie s or

audiences

 a major expansion of the range and the depth of research and its application

 outstandingly novel, innovative and/or creative.

b. In assessing work as being three star (quality that is internationally excellent in terms

of originality, significance and rigour but which falls short of the highest standards of

excellence), sub-panels will expect to see evidence of, or potential for, some of the

following types of characteristics across and possibly beyond its area/field:

 an important point of reference

 of considerable influence

 a catalyst for, or important contribution to, new thinking, practices, parad igms,

policies or audiences

 a significant expansion of the range and the depth of research and its application

 significantly novel or innovative or creative.

c. In assessing work as being two star (quality that is recognised internationally in

terms of originality, significance and rigour), sub-panels will expect to see evidence

of, or potential for, some of the following types of characteristics across and possibly

beyond its area/field:

 a recognised point of reference

 of some influence

 an incremental and cumulative advance on thinking, practices, paradigms,

policies or audiences

 a useful contribution to the range or depth of research and its applicat ion.

49

d. In assessing work as being one star (quality that is recognised nationally in terms of

originality, significance and rigour), sub-panels will expect to see evidence of the

following characteristics within its area/field:

 an identifiable contribution to understanding without advancing existi ng paradigms

of enquiry or practice

 of minor influence.

e. A research output will be graded ‘unclassified’ if it is either:

 below the quality threshold for one star; or

 does not meet the definition of research used for the REF. (See ‘Guidance on

submissions’, Annex C).

Output types

206. The main panels welcome all forms of research output that fulfil the eligibility criteria for the

REF (set out in Part 3, Section 2 of ‘Guidance on submissions’). All forms of output, in any language,

will be considered equitably, with no distinction being made between the type of research or form of

output submitted. The sub-panels will neither advantage nor disadvantage any ty pe of research or

form of output. The main panels encourage submitting institutions to refer to the glossary of output

types for information on the categories under which outputs may be submitted for assessment (see

‘Guidance on submissions’, Annex K).

207. No sub-panel will use journal impact factors or any hierarchy of journals in their assessment of

outputs. No output will be privileged or disadvantaged on the basis of the publisher, where it is

published or the medium of its publication.

208. Reviews, textbooks and edited works (including editions of texts) and translations may be

included if they embody research as defined in ‘Guidance on submissio ns’, Annex C. Editorships of

journals and other activities associated with the dissemination of research findings should not be

listed as an output on REF2.

209. Each submitted output needs to have a single classification selected from the list of eligible

output types (see ‘Guidance on submissions’, Annex K). The pur pose of the classification is to assist

in the management of the collection and distribution of outputs, the allocation of outputs to expert

reviewers, and a post-submission analysis of types of outputs submitted. The sub-panel will assess

the research content of the material submitted regardless of the classification.

50

Main Panel D supplementary criteria – output types

210. It is not unusual for an output submitted to the sub-panels in Main Panel D to encompass a

number of different output types, such as a ‘design’ output which includes a journal article and a

patent application; or an ‘artefact’ or prototype that has been the subject of an exhibition; or a data

set or database which includes critical insight or analysis; or a ‘composition’ t hat has also been a

performance or recording; or an exhibition where the research may be curatorial (or involve or

support co-curation) and/or evident in the development of the interpretative strategy , exhibition

text/narrative or catalogue. Submitting institutions should select a single output type, and the

panel will judge the research content of the material submitted regardless of the classification.

211. An additional classification of ‘Translation’ has been added to the list of output types, for the

submission of works of translation of literary or scholarly texts or other cultural documents that

constitute original, significant and rigorous research. Translations that meet the definition of

research will often exhibit a deep insight into the source material, while drawing on and reflecting

specialist knowledge of its historical, political, social and cultural contexts, and will also rely on a

detailed engagement with style in both the source and target languages. Research may, as a

result, be reflected in the critical apparatus associated with a translated text but will also be

inherent in the translation process itself. Such outputs will often contribute to the development and

maintenance of intellectual infrastructure of subjects and disciplines. They may demonstrate

research practice that is critical and/or creative, and may also serve as substantial interv entions in

intellectual and cultural life in their own right.

212. It is also anticipated that commentaries will be submitted, if they embody research as

defined for the purposes of REF. Like some translations, commentaries often include research

that encompasses work with original manuscripts, textual criticism, the historic al, political, social

and cultural context of a text, its history of reception and influence, and issues in its contemporary

interpretation. Commentaries should be submitted under the “output type” (list ed in ‘Guidance on

submissions’, Annex K) which fits their form of publication, most commonly “aut hored book”.

213. For indicative guidance on what material to include in the submission, please refer to the

table of output types in Annex C and the summary of ‘Additional Informatio n’ in Annex B.

Outputs with significant material in common

214. As stated in ‘Guidance on submissions’ (paragraph 220), where two or more research outputs

within a submission include significant material in common, the sub-panels w ill assess each output

taking account of the common material only once. Where a sub-panel judges that they do not

contain sufficiently distinct material and should be treated as a single output, an unclassified score

would be given to the ‘missing’ output.

51

215. Where a submitted output includes significant material in common with an output submitted to

REF 2014, as stated in ‘Guidance on submissions’ (paragraph 259), subm issions should explain

how far the earlier work was revised to incorporate new material (maximum 100 words).

Co-authored outputs

216. As set out in the ‘Guidance on submissions’, for co -authored outputs:

 The number of other authors will be required.

 Where two or more co-authors or co-producers of an output are returned in different

submissions (whether from the same HEI or different HEIs), the output may be listed in any

or all of these submissions.

 In exceptional cases, co-authored outputs may be submitted up to two times within a

submission. This applies only to submissions made to Main Panel D (paragraphs 233 to

235).

 Where there are substantial pieces of co-authored work, reflecting large-sc ale or intensive

collaborative research within the same submitting unit, and a double -weighting request has

been submitted for the output, institutions may attribute the output to a maximum of two

members of staff returned within the same submission. This output may be counted as the

required minimum of one for each staff member. The inclusion of any reserv e outputs in this

instance must be in accordance with the minima and maxima requirements where the panel

does not accept the request for double-weighting.

217. Institutions may only attribute co-authored outputs to individual memb ers of staff who made a

substantial research contribution to the output. The main panels set out below t heir requirements for

information about the author’s contribution.

218. Where information is requested and a sub-panel judges that the staff mem ber to whom the

output is attributed has not made a substantial research contribution to a co-authored output , the

sub-panel will grade that occurrence of the output as ‘unclassified’.

219. Information may also be requested through an audit to verify that an author made a substantial

research contribution to the output. Where this cannot be verified the output will be graded as

‘unclassified’.

220. Once a sub-panel has determined that each co-author’s contribut ion to the research content of

the output is distinct and substantial, it will assess the quality of the output as a whole, taking no

further regard of each individual co-author’s contribution.

52

Main Panel A supplementary criteria – co-authored outputs

221. An output may only be submitted once in a UOA submission by any given HEI. Where co-

authors represent different UOAs within an HEI, the output can be submitted to each UOA. Where

co-authors come from different institutions the output can be submitted by each HEI.

Information required about the author’s contribution

222. No additional information is required in form REF2 about t he author’s contribution to co-

authored outputs where either:

 there are 15 authors or fewer; or

 there are more than 15 authors but the submitted member of staff to whom the output is

attributed is identified as either lead or corresponding author (regardless of the number of

authors).

223. Whether first author, last author, alphabetical or some other order , Main Panel A considers

that the lead and corresponding authors should be easily identifiable within the submitted output.

Provided the submitted member of staff is clearly identifiable within the output as lead or

corresponding author, including any instances of where that role may be shared, no additional

information is required.

224. For each submitted co-authored output where there are more than 15 authors and where

the submitted member of staff is not identified as the lead or corresponding a uthor, institutions are

required to affirm the substantial contribution to the research by the submitted member of staff.

This should be done by entering the following statements in REF2, including at least one element

from each of a and b:

a. The author made a substantial contribution either to the conception and design of the

study; or to the organisation of the conduct of the study; or to carrying out the study;

or to analysis and interpretation of study data.

And

b. The author helped draft the output; or critique the output for important intellectual

content.

225. Where the author contribution has been included in the output acknowledge ments, this will

take precedence to the statement on co-authored outputs. Statements on author contr ibution will

be subject to audit.

53

Main Panel B supplementary criteria – co-author contribution

226. An output may only be submitted once in a UOA submission by any given HEI. Where co-

authors represent different UOAs within an HEI, the output can be submitted to each UOA. Where

co-authors come from different institutions the output can be submitted by each HEI.

Additional requirement for information on co-authored outputs

Sub-panels 7, 8, 10, 11 and 12

227. The sub-panels do not require the submission of information about the individual co-

author’s contribution to a co-authored output and, if received, will take no acc ount of such

statements.

Sub-panel 9 only

228. No additional information is required in form REF2 about t he author’s contribution to co-

authored outputs where either:

 there are 15 authors or fewer; or

 there are more than 15 authors but the submitted member of staff to whom the output is

attributed is identified as either lead or corresponding author (regardless of the number of

authors).

229. Whether first author, last author, alphabetical or some other order , Sub-panel 9 considers

that the lead and corresponding authors should be easily identifiable within the submitted output.

Provided the submitted member of staff is clearly identifiable within the output as lead or

corresponding author, including any instances of where that role may be shared, no additional

information is required.

230. For each submitted co-authored output where there are more than 15 authors and where

the submitted member of staff is not identified as the lead or corresponding a uthor, institutions are

required to affirm the substantial contribution to the research by the submitted member of staff.

This should be done by entering up to 100 words in which the author contribution is articulated.

231. Where the author contribution has been included in the output acknowledge ments, this will

take precedence to the statement on co-authored outputs. Statements on author contr ibution may

be subject to audit.

54

Main Panel C supplementary criteria – co-authored outputs

231. An output may only be submitted once in a UOA submission by any given HEI. Where co-

authors represent different UOAs within an HEI, the output can be submitted to each UOA. Where

co-authors come from different institutions the output can be submitted by each HEI.

Additional requirement for information on co-authored outputs

232. The sub-panels in Main Panel C do not require the submission of information about the

individual co-author’s contribution to a co-authored output and, if received, will take no account of

such statements. The sub-panels may seek to verify a contribution via audit in accordance with

paragraph 219.

Main Panel D supplementary criteria – co-author contribution

233. Exceptionally, the sub-panels in Main Panel D will accept the inclusion of the same co-

authored output up to two times in a submission. This provision is in recognition of the constraints

to the size of the output pool as a result of a combination of factors, including pub lication patterns

in Main Panel D; that many submissions will be from small departments; and that a number of

UOAs in Main Panel D are comprised of a broad spectrum of sub-disciplines, many of which will

be in separate departments in submitting HEIs.

234. Such outputs should not account for more than five per cent of the outputs (or one output,

whichever is the greater) within a submission. Alternatively, where such outputs satisfy the

requirements for double-weighting, submitting HEIs should use the provision outlined in the final

bullet point in paragraph 216 instead – no quota applies in that case. These t wo provisions cannot

be used in combination.

235. Consequently, a co-authored output can be submitted:

 once as a single output; or

 twice, attributed to two of the authors when it satisfies the criteria for double-

weighting (see paragraph 216); or

 twice, attributed to two of the authors, within the quota of five per cent or one output

(whichever is the greater), when it does not meet the double-weighting criteria.

Additional requirement for information on co-authored outputs

236. The sub-panels in Main Panel D do not require the submission of information about the

individual co-author’s contribution to a co-authored output and, if received, will take no account of

such statements. The sub-panels may seek to verify a contribution via audit in accordance with

paragraph 219.

55

Double-weighted outputs

237. The main and sub-panels recognise that there will be cases where the scale of academic

investment in the research activity and/or the intellectual scope of the research output is

considerable. The main and sub-panels want to recognise and double-weight such outputs in the

assessment, so that they will count as two outputs both in a submission and in the calculation of the

outputs sub-profile. The main panels have set out below their expectations in relation to receiving

requests for double-weighting.

238. Institutions’ requests for double-weighting must be accompanied by a statement of up to 100

words explaining how the output satisfies the criteria.

239. As set out in the ‘Guidance on submissions’ (paragraphs 282 to 283), a reserve output may be

submitted where a request for double-weighting is made. The reserve output may be attributed to

any submitted member of staff, providing that it is in accordance with the minima and maxima

requirements for attributing outputs to staff.

240. Sub-panels will double-weight an output only if a request is made by the submitting institution,

and the case is accepted by the sub-panel. Sub-panels will not double-weight any output for which a

request has not been made by the institution.

241. Sub-panels will assess the claim for double-weighting separately from assessing the quality of

the output, and there is no presumption that double-weighted outputs will be assessed at higher-

quality grades. When assessing claims for double-weighting, the su b-panel will not privilege or

disadvantage any particular form of research or type of output.

Main Panels A and B supplementary criteria – double-weighting

242. The sub-panels anticipate that they will double-weight outputs only where they derive from

substantial academic endeavour by the member of staff against whom the out put is listed in the

submission. Such endeavour might be understood in terms of (but is not limit ed to) the ambition of

the project.

243. Considering the patterns of publication across Main Panel A and B ’s areas of activity, the

sub-panels expect that such requests will occur only exceptionally. In particular , the sub-panels

anticipate that outputs published as journal articles and conference papers will not normally

embody work of this nature, and they therefore do not normally expect to receiv e requests for

double-weighting these types of outputs.

56

Main Panels C and D supplementary criteria – double-weighting

244. The sub-panels strongly encourage submission of outputs of extended sca le and scope for

consideration as double-weighted outputs.

245. The submission of a statement to evidence the claim for double-weighting is required and

should briefly outline the reasons for the request, addressing the characteristics below.

246. The sub-panels in Main Panels C and D have identified the following charact eristics which

might apply (individually or in combination) to the research effort associated with a double-

weighted output:

 the production of a longer-form output (e.g. book, long-duration creative work or multi-

component output) demonstrating sustained research effort

 the generation of an extended or complex piece of research

 the collection and analysis of a large body of material

 the use of primary sources which were extended, complex or difficult to access

 the presentation of a critical insight or argument which was dependent upon the

completion of a lengthy period of data collection or investigation of materials

 the undertaking of a complex, extended and/or multi-layered proc ess of creative

investigation (individual or collective)

 the investigation of a given theme in considerable depth, fro m different perspectives,

and/or in relation to different contexts.

It is recognised that in some instances the characteristics listed in paragra ph 246 may apply to

short-form outputs such as journal articles, book chapters and short-durat ion creative work and

justify the double-weighting of such items.

247. It is expected that most books, monographs, novels or longer-form outputs warrant double-

weighting, although claims will not automatically be accepted.

Additional information for outputs

248. The requirements for additional supporting information for each main pane l are set out below.

A summary of additional information for outputs required by the main panels is set out in Annex B.

57

249. HEIs are instructed to ensure that additional supporting information for outputs is succinct,

verifiable, and externally referenced where appropriate. No other information shou ld be included,

and sub-panels will take no account of any such information if submitted.

250. Where additional information is accepted, the information provided mu st not include citation

data or journal impact factors. Any panels that make use of citation data will be provided with the

data by the REF team. Sub-panels will take no account of any citation data pr ovided directly by the

HEI. No sub-panel will use journal impact factors or any hierarchy of journals in their assessment of

outputs.

251. For research outputs in languages other than English (see ‘G uidance on submissions’,

paragraphs 285 to 287) submitted to all sub-panels, a short abstract (up t o 100 words) in English

should be provided to describe the content and nature of the work. This abstract does not form part

of the assessment of the submitted output. See below for an exception applyin g to Sub-panel 26

(Modern Languages and Linguistics).

Main Panel A supplementary criteria – additional information for outputs

Information about the research process and/or content

252. For non-text, or practice-based outputs (including patents, software and standards

documents), all sub-panels welcome the submission of a description in REF2 of the research

process and research content, where this is not evident within the output (maximum 300 words),

as described in ‘Guidance on submissions’ (paragraph 284.a.).

Factual information about significance

253. The sub-panels do not wish to receive additional information abo ut the significance of

outputs (‘Guidance on submissions’, paragraph 284.b.) and, if received, wil l take no account of

any statement beyond those that have been requested by Main Panel A, as summaris ed in Annex

B.

Main Panel B supplementary criteria – additional information for o utputs

Information about the research process and/or content (Sub-panels 7 to 12)

254. For non-text, or practice-based outputs (including patents, software and standards

documents) all sub-panels require the submission of a description of the research proces s and

content, where this is not evident within the output (maximum 300 words).

58

255. For reviews, sub-panels welcome the identification of the original researc h or new insights

reported, to assist with the assessment of research quality (maximum 300 words).

Factual information about significance – Sub-panels 11 and 12 only

256. Sub-panels 11 and 12 consider that the nature of their disciplines is such that the

significance of an output may not be fully evident within the output itself. They therefore invite

factual information to be provided (maximum 100 words) that could include, for example,

additional evidence about how an output has gained recognition, impacted the state of the art, led

to further developments, or has been applied.

257. HEIs are instructed to ensure that such evidence is succinct, verifiab le, and externally

referenced where appropriate. Where claims are made relating to the industrial significance of the

output, the name and contact details of a senior industrialist must be given to allow verification of

claims. Information provided should not comprise a synopsis of the output, a volunteered opinion

as to the quality of the output or citation data, and information provided that is of this nature will be

disregarded. It is expected that, in most cases, sufficient information will be provided in

significantly fewer words than the 100-word limit.

Allocation of outputs for assessment

258. The following sub-panels request information to assist i n allocating outputs to

appropriate readers during the assessment phase in 2021. The information will not be used for

any other purpose:

 Sub-panels 7, 11 and 12 will provide subject-specific taxonomies. Submitting units will be

asked to identify the topics relevant to submitted outputs using the chosen taxonomy .

 Sub-panel 10 will request up to two keywords which will categorise the topic(s) covered by

the output. The keywords may be taken from the MSC2020 taxonomy where appro priate

or may be provided by the submitting UoA. Submitting units are asked to be as consistent

as possible in use of their chosen keywords within their submission.

Main Panel C supplementary criteria – additional information for outputs

Information about the research process and/or content

259. For any submitted outputs where the research content and/or process is not evident from

the output, such as non-text outputs or teaching materials, submissions should include a

statement which identifies the research questions, methodology and means of dissemin ation

(maximum 300 words).

59

Submission of practice-based outputs

260. To ensure that practice- based outputs are assessed on an equal basis with other outp uts,

submissions should include an explan atory presentation of the buil ding, design or intervention in

an easily-handled paper-based for mat (for example, a PDF which cou ld include photographs,

figures or diagrams) sufficient to al low the panel both to understand th e output without visiting it,

and to make a judgement of its r esearch contribution.

261. For software and data sets, a full written description should be provided in a paper-based

format in order to avoid accessib ility problems, including details of how and where the data set or

software can be accessed. Such a ccess should preserve the anonymi ty of the reviewer.

262. Where the form of an output makes this essential, the paper-base d submission may be

supplemented by limited visual mater ial in an accessible format such as a video file.

Main Panel D supplementary criteria – additional information for outputs

Submission of research outputs to Main Panel D

263. The table attached in Annex C indicates the classification of output types and Main Panel

D’s guidance on the content of submissions relating to each output type.

264. In all cases where the role of the researcher, or the research process, is not evident within

the submitted output, submitting units are strongly encouraged to submit a statement of up to 300

words.

265. For outputs where the role of the researcher or the research process is not evident in the

submitted output, submitting units have the following options in choosing how best to present the

output (irrespective of the classification):

 As a single item, with a 300-word supporting statement.

 As a multi-component output, with a 300-word supporting state ment.

 As a single item, supported by contextual information (prev iously called a ‘portfolio’). Both

the item and the contextual information may include moving image, sonic, visual or other

digital media or written text, as appropriate, to enable the panel to access the research

dimensions of the work and to assess its significance, originality and rigour. The 300-word

statement should be used to indicate what is the output and what is the contextual

information.

60

 As a multi-component output, supported by contextual information (previously called a

‘portfolio’). Both the output and the contextual information may include movin g image,

sonic, visual or other digital media or written text, as appropriate, to enable the panel to

access the research dimensions of the work and to assess its significance, originality and

rigour. The 300-word statement should be used to indicate what is the output and what is

the contextual information.

266. The entirety of the material submitted (the output and the 300-word state ment where

provided) should provide the panel with coherent evidence of the research dimensions of the work

in terms of:

 the research process – the question a nd/or issues being explored, the process of

discovery, methods and/or methodologies, the creative and/or intellectual context or

literature review upon which the work draws, or challenges or critiques

 the research insights – the findings, discov eries or creative outcomes of that process

 the dissemination – how and where the insights or discoveries were ‘effectively shared’ .

This needs to satisfy the REF requirements around the dates at which work first entered

the public domain (‘Guidance on submissions’, paragraph 205.b.). The principle that no

output will be privileged or disadvantaged on the basis of the publisher, where it is

published or the medium of its publication (paragraph 217), will also apply in relation to the

broad range of modes through which practice outputs enter the public domain.

267. The bullet points above are derived from the REF definition of research as set out in the

‘Guidance on submissions’, Annex C. They are intended to assist HEIs by providing a flexible

framework for the succinct and coherent presentation of the output. The objective is to enable

panels to assess the originality, significance and rigour of the research. The sub-panels will ignore

any additional material that includes evaluative commentary on the perceived quality of a

research output.

268. The format for the presentation of outputs (irrespective of their classificat ion) is flexible,

within the requirement to limit the format either to an electronic sub mission which is submitted via

the REF submission system either as a URL, Digital Object Identifier (DOI), or by uploading a

PDF; or as physical material which is sent to the RE F team, and which may include

digital/electronic material on a media storage device e.g. USB, CD. An individual output cannot be

submitted both electronically via the REF submission system and as a physical outp ut.

269. There will be many outputs that will meet the REF definit ion of research as “a process of

investigation, leading to new insights effectively shared” without the need for additional

information, and these may include examples of creative practice. Where the research process is

not self-evident, the guidance in paragraphs 264 to 267 should be followed.

61

Statement on research contribution

270. Sub-panels expect to receive anthologies, edited books, special issues of journals and

curatorial projects where the researcher has made a demonstrable contribution to the research

published (in addition to any chapter published in the same work). Where such a research

contribution is part or all of the output to be assessed, the whole work should be submitted.

Submitting units may provide a statement (of up to 300 words) to clarify the nature of the

individual’s research contribution.

Statement on rationale for grouping short items

271. Substantial dictionary or encyclopaedia entries and groups of short items including for

example portfolios of creative writing, or related critical works (where such work em bodies

research as defined for the purposes of the REF in ‘Guidance on submissio ns’), may be submitted

as a single output, along with an explanation of the rationale for grouping such ite ms (of up to 300

words).

Information for outputs in languages other than English

272. For research outputs in languages other than English (‘Guidance on submissions’,

paragraphs 285 to 287), a short abstract (of up to 100 words) in Eng lish should be provided to

describe the content and nature of the work. This abstract does not form part of the assessment

of the submitted output. This requirement is waived for outputs submitted in UOA 26 (Modern

Languages and Linguistics) if the output is produced in any of the languages within the remit of

that UOA: that is, all Celtic, Slavonic, Germanic and Romance languages.

Allocation of outputs for assessment

273. The following sub-panels request information to assist i n allocating outputs to

appropriate readers during the assessment phase in 2021. The information will not be used for

any other purpose:

 Sub-panels 26, 27, 28, 29, 33 and 34 will provide subject-spec ific taxonomies. Submitting

units will be asked to identify the topics relevant to submitted outputs using the chosen

taxonomy.

Citation data

274. The main panels set out below which of the sub-panels will use cit ation data. Where sub-

panels use citation data, the following criteria apply:

62

a. Where available and appropriate, citation data will be considered as an indicator of the

academic significance of the research output. This will only be one element to inform

peer-review judgements about the quality of the output, and will not be u sed as a

primary tool in the assessment.

b. The absence of citation data for an output will not be taken to mean an absence of

academic significance.

c. The sub-panels recognise that the citation count is sometimes, but not always, a reliable

indicator. They are also aware that such data may not always be available, and the level

of citations can vary across disciplines and across UOAs. Sub-panels will be mindful

that citation data may be an unreliable indicator for some forms of output (for examp le,

relating to applied research), and the limitations of such data for outputs in languages

other than English and for recent outputs. Sub-panels will take due regard of the

potential equalities implications of using citation data, as outlined in the ‘ Equality briefing

for REF panels’ (2018/05).

d. Sub-panels will use citation data only where provided by the REF team, and will not

refer to any additional sources of bibliometric analysis, including in particular journal

impact factors and other journal rankings.

275. Those panels using citation data will do so with in the framework set out in ‘Guidance on

submissions’ (paragraphs 288 to 292). Panels wi ll continue to rely on expert review as the primary

means of assessing outputs, in order to reach rounded judgements about the full range of

assessment criteria (‘originality, significance and rigour’). They will also recognise th e significance of

outputs beyond academia wherever appropriate, and will assess all outputs on an equal basis,

regardless of whether or not citation data is available for them.

276. The panels using citation data will receive guidance from the Forum for Responsible Research

Metrics to ensure that they are used appropriately.

Main Panel A supplementary criteria – citation data

277. All sub-panels in Main Panel A will use citation data, where appropriate an d available, as a

potential indicator of academic significance to inform the assessment of outp ut quality.

Main Panel B supplementary criteria – citation data

278. Sub-panels 7, 8, 9 and 11 acknowledge that citation data are widely used and consider that

they are well understood in the disciplines covered in their UOAs. These sub-panels w ill receive

63

citation data, where available, and may make use of the data as part of the indication o f academic

significance to inform their assessment of output quality.

279. Sub-panels 10 and 12 believe that citation data in their disciplines cannot be used to

provide sufficient added value to inform the assessment of output quality. Th ey therefore will not

receive nor make use of citation data.

Main Panel C supplementary criteria – citation data

280. Sub-panel 16 (Economics and Econometrics) will receive citation data, where available, and

will make use of the data supplied by the REF team where it is considered ap propriate as an

additional piece of supplementary evidence to support the initial assessment of outputs, not as a

determining factor. Sub-panel 16 will take account of the well-known limitations of citations,

including equality, diversity and inclusion issues.

281. The remaining sub-panels within Main Panel C will neither receive nor make use of citation

data.

Main Panel D supplementary criteria – citation data

282. The sub-panels in Main Panel D will neither receive nor make use of citation data.

Section 4: Impact

Introduction

283. This section should be read alongside ‘Guidance on submiss ions’ Part 3, Section 3, which sets

out the generic definition of impact for the REF, the requirements for submitting impa ct case studies,

the associated eligibility guidelines, and the generic assessment criteria and level definitions. The

sub-panels will assess impact in accordance with this framework.

284. The main and sub-panels have determined that no one model or relationsh ip will be

considered intrinsically preferable, and each impact case study will be assessed on its own merits.

285. In drawing up their assessment criteria and the advice to submitting instituti ons, the main

panels strongly advise institutions that the guidance provided here, particularly regardi ng examples

of impacts and evidence and/or indicators for those impacts, should not be read as exhaustive,

prescriptive or limiting either for institutions or panels. They also recogn ise that the examples

provided in Table 1 (Annex A) may fit under headings other than those to w hich they have been

64

presented. The main panels wish to encourage the submission of a broad range of types of impact,

as evidence of the strength and diversity of the impact of research across all disciplines, and

anticipate that extremely strong impact case studies will be submitted which do not relate to any of

the examples provided in the guidance. The examples are offered to assist ins titutions, not to

constrain them.

286. The panels also acknowledge that there are multiple and diverse pathw ays through which

research achieves impact. Impact may be the result of individual or collective researc h (or a

combination of these) within or between a range of organisations, within HE and beyond, including

collaboration beyond the UK. The associated impact may be achieved by a v ariety of possible

models: from individuals, to inter-institutional groups, to groups including both acade mic and non-

academic participants. The relationship between research and impact can be indirect or non-linear.

The impact of research may be foreseen or unforeseen. It can emerge as an end product, but can

also be demonstrated during the research process. Impact takes place through a wide variety of

mechanisms. It may effect change or enrichment for local, national or internatio nal communities,

groups or individuals. Consequently, public engagement may be an important feature of many case

studies, as the mechanism by which the impact claimed has been achieved.

Impact criteria

287. The sub-panels will assess the ‘reach and significance’ of impacts on the economy, society

and/or culture that were underpinned by excellent research conducted in the submitted unit,

according to the generic criteria and level definitions. This section provides a descriptiv e account of

how the sub-panels will interpret and apply the generic criteria for assessing i mpact.

288. Reach will be understood as the extent and/or diversity of the beneficiaries of the impact, as

relevant to the nature of the impact. Reach will be assessed in terms of the extent to which the

potential constituencies, number or groups of beneficiaries have been reached; it will not be

assessed in purely geographic terms, nor in terms of absolute numbers of beneficiaries. The criteria

will be applied wherever the impact occurred, regardless of geography or location, and whether in

the UK or abroad.

289. Significance will be understood as the degree to which th e impact has enabled, enriched,

influenced, informed or changed the performance, policies, practices, products, services,

understanding, awareness or wellbeing of the beneficiaries.

290. The sub-panels will make an overall judgement about the reach and significance of impacts,

rather than assessing each criterion separately. While case studies n eed to demonstrate both reach

and significance, the balance between them may vary at all quality levels. The sub-panels will

exercise their judgement without privileging or disadvantaging either reach or significance.

291. HEIs may submit case studies describing impacts at any stage of development or maturity.

However, the assessment will be solely on the impact achieved during the assessme nt period,

65

regardless of its stage of maturity. No account will be taken of anticipated or future potential impact,

nor of impact that occurred outside the assessment period (1 August 2013 to 31 July 2020).

Continued impact case studies

292. As set out in the ‘Guidance on submissions’ (para graphs 314 to 317), case studies cont inued

from examples submitted in 2014 will be eligible for submission in REF 2021. All impact case studies

submitted in REF 2021 must meet the same eligibility criteria, including the lengt h of the window for

underpinning research (1 January 2000 to 31 December 2020) and the assessment peri od (1 August

2013 to 31 July 2020) for the impact described. The main panels set out below t heir expectations in

relation to receiving continued case studies in the assessment.

293. The panels encourage submitting units to submit their strongest case studies irrespective of

whether they are new examples or represent continuing impact from those submitt ed in REF 2014.

Main Panel A supplementary criteria – continued case studies

294. Main Panel A will assess each case study on merit and wishes to receive information on

how any continued case study relates to that submitted in REF 2014. Panel members wi ll have

access to the REF 2014 database

2

and may refer to this to understand the context of the 2021

case study.

Main Panels B, C and D supplementary criteria – continued case studies

295. The sub-panels will assess each case study on merit and do not wish to receive information

on how any continued case study relates to that submitted to REF 2014. If any such information is

provided, the sub-panels will not take it into account during the assessment process.

Range of impacts

296. The main panels welcome case studies that describe any type(s) of impact which fulfil the

definition of impact for REF (see ‘Guidance on submissions’, paragraphs 297 to 302). They

acknowledge that impact may take many forms and occur in a wide range of s pheres. They welcome

case studies which describe impacts that have provided benefits to one or more areas of the

economy, society, culture, public policy and services, health, production, environment, inter national

development or quality of life. The panels will also welcome impacts that descr ibe changes or

benefits resulting from research that leads to a decision not to und ertake a particular course of

2

https://impact.ref.ac.uk.

66

action. Sub-panels recognise the value of co-produced impact and expect instit utions to ensure case

studies of this nature clearly acknowledge the work of partner organisations .

297. Impacts can be manifested in a wide variety of ways including, but not limited to: the many

types of beneficiary (individuals, organisations, communities, industry, regions and other entities);

impacts on products, processes, behaviours, policies, practices and understanding; and avoidance

of harm or the waste of resources in the widest sense. Impact of any type may be local, regional,

national or international, in any part of the world.

298. Research may underpin impact which provides benefits in more than one area. An impact

case study may therefore describe more than one type of impact arising from such bodies of work;

for example, a new drug can generate both health and economic impact, a new energy technology

can generate both environmental and economic impact, and a new exhibition or performance can

generate cultural, economic and social benefits.

299. Engaging the public with the submitting unit’s research (for example, through citizen science,

patient and public involvement in health, or through public and community engagement), is an

activity that may lead to impact. Sub-panels will welcome, and assess equitably, case studies

describing impacts achieved through public engagement, either as the main impact described or as

one facet of a wider range of impacts. Panels expect that case studies based on public engagement

will demonstrate both reach (for example, through audience or participant figures and demographics)

and significance, and will take both into account when assessing the impacts. Examples of impacts

arising from public engagement can be found as part of Table 1 (Annex A).

300. Examples are provided in Table 1 (Annex A) as a guide to the range of potential impacts that

may be eligible as case studies. The list is not exhaustive or exclusive, and does not rank examples

in any way. In making use of this to assist with the preparation of submissions, HEIs should note

that:

a. The list of types and examples of impacts is not intended to be exhaustive, and some

examples are relevant to more than one type of impact. Sub-panels wish to encourage

HEIs to submit case studies describing any impacts that meet the generic definition in

‘Guidance on submissions’, paragraphs 297 to 302.

b. HEIs are not required to align submitted case studies specifically with the particular

types of impact defined in the list.

301. Impacts on or through teaching within and beyond the submitting institutio n may be submitted,

and examples are included in Table 1 (Annex A). Where impact on teaching within th e submitting

unit’s own institution is included in a case study, sub-panels will give cons ideration to the following:

 the reach of the impact, that is the extent or divers ity of the communities affected by the

change to teaching practice

67

 the significance of the impact, that is, the extent to which teaching practice was enriched,

influenced or changed at the organisation(s) involved and/or the extent to which individuals

experiencing changed teaching practice were enriched, influenced or changed.

302. Sub-panels expect that impact on teaching within the submitting u nit’s own institution may

most convincingly form a component of a wider case study that also includes impact s beyond the

institution.

303. The panels acknowledge that there may be impacts arising from research which take forms

such as holding public or private bodies to account or subjecting propose d changes in society, public

policy, business practices, and so on to public scrutiny. Such holding to account or public scrutiny

may have had the effect of a proposed change not taking place; there may be circumstances in

which this of itself is claimed as an impact. There may also be examples of research findings having

been communicated to, but not necessarily acted upon, by the intended audience, but which

nevertheless make a contribution to critical public debate around policy, social or business issues.

The panels also recognise that research findings may generate critique or d issent, which itself leads

to impact(s). For example, research may find that a government approach to a particular social,

health, food-/biosecurity or economic issue is not delivering its objectives, which leads to the

approach being questioned or modified.

Evidence of impact

304. Each case study must provide a clear and coherent narrative that includes an account of who

or what audiences, constituencies, groups, organisations, places, publics, sectors and so on, have

benefited, been influenced, or acted upon. The sub-panels will use their expert judge ment regarding

the integrity, coherence and clarity of the narrative of each case study, but will expect the key claims

made in the narrative to be supported by evidence and indicators.

305. In assessing impact case studies, sub-panels will consider both the evidenc e linking excellent

research and bodies of work within the submitting unit to the impact(s) claimed, recogn ising that this

relationship can be indirect or non-linear, and the evidence of the reach and significance of the

impact. Within their narrative account in the case study, submitting un its should provide the

indicators and evidence most appropriate to support the impact(s) claimed. Where using quantitative

indicators, institutions should follow the guidance on their standardised presentation, av ailable at

www.ref.ac.uk, under Guidance.

306. Submitting units should focus on providing evidence of the impacts ac hieved, as distinct from

evidence of dissemination and uptake, in order to demonstrate both the reach and significance of the

impact(s) claimed (see paragraph 290). For example, attendance figures at an event may illustrate

the pathway to a change in understanding or awareness and provide an indic ation of the reach of

the impact. However, on their own, they would not serve as evidence of the significance of the

impact, which might be demonstrated, for example, through participant feedback or critical reviews.

68

307. Submitting units should ensure that, so far as possible, any evidence cite d is independently

verifiable. Verifiable sources for key evidence and indicators should be provided in Section 5 of the

impact case study template and the relevant evidence provided to the REF team as set out in the

‘Guidance on submissions’ Part 3, Section 3.

308. The main panels recognise that some evidence in case studies may be of a confidential or

sensitive nature. The arrangements for submitting and assessing case studies that include such

material are set out in ‘Guidance on submissions’, paragraphs 307 to 308.

309. The panels anticipate that impact case studies will refer to a wide range of types of evidence,

including qualitative, quantitative and tangible or material evidence, as approp riate. Individual case

studies may draw on a variety of forms of evidence and indicators. The panels do not wish to pre-

judge forms of evidence. They encourage submitting units to use evidence most appropr iate to the

impact claimed. A diversity of evidence is welcome, and no type of evidence is inherently preferred

over another. Cited evidence should provide a convincing and verifiable link betw een the

underpinning research or bodies of work and the impact claimed, as well as convinc ing and

verifiable evidence of the reach and significance of the impact.

310. Testimonials should draw on statements of fact and relate specifically to the impact(s) claimed.

There may be occasions where opinion-based testimonials are appropriate (for examp le, where the

impact is on public understanding of an issue). Where such testim onials are cited as evidence in

impact case studies, sub-panels will consider the extent to which the testimonial citat ion evidences

the significance of the claims. Sub-panels recognise the varying degrees to which eviden ce and

indicator information may be available to HEIs. Where testimony is cited, it should be made clear

whether the source is a participant in the process of impact delivery (and the degree to which this is

the case), or is a reporter on the process.

311. Where corroborating evidence is reviewed as a consequence of audit, it will be used solely to

verify the claims made about the impact. Additional information included in such ev idence will not be

used to supplement or strengthen the impact case study narrative.

312. The examples in Table 1 (Annex A) provide a guide to potentia l types of evidence or indicators

that may be most relevant to each of the broad areas of impact described i n Table 1. However,

institutions should note that:

 This is not intended to be exhaustive.

 Some indicators may be relevant to more than one type of impact.

 Sub-panels will consider any relevant, verifiable evidence.

69

Underpinning research

313. Sub-panels need to be assured that the impact claimed is based on research at least

equivalent to two star, as defined in ‘Guidance on submissions’, Annex A. Submitting un its are

required to identify the underpinning research and provide in Section 3 up to six key references to

research produced by the submitting unit in the period 1 January 2000 to 31 December 2020 that

underpins the impact described in the case study. The sub-panels will not expect each referenced

item to meet the quality threshold, but will wish to be satisfied that the research as a whole was of at

least two-star quality.

314. Underpinning research may be a body of work produced over a number of years, within the

REF timeframe (1 January 2000 to 31 December 2020) or may be the output( s) of a particular

project. It may be produced by one or more individuals. Underpinning research out puts may include

the full range of types listed in the output glossary (‘Guidance on submissio ns’, Annex K) and are not

limited to printed academic work. They may include, but are not limited to: new materials, devices,

images, artefacts, products and buildings; confidential or technical reports; intellectual property,

whether in patents or other forms; performances, exhibits or events; and w ork published in non-print

media. All forms of output cited as underpinning research will be considered on an equal basis, with

no distinction being made between the types of output referenced.

315. Provided the sub-panel is satisfied that the quality threshold has been met, the quality of the

underpinning research will not be taken into consideration as part of the assessment of the reach

and significance of the claimed impact.

316. Underpinning research referenced in a case study may also be included in a submission as an

output (listed in REF2), without disadvantage. In these situations, the assessment of the impact case

study will have no bearing on the assessment of the quality of the output. The assessment of the

quality of the output may inform the assessment of the case study, only in terms of assuring the

threshold for underpinning research quality.

317. The institution submitting a case study must have produced research which has made a

distinct and material contribution to the impact described in the case study. Sub-pan els will expect to

see clear narrative evidence of this in the case study. The panels recognise that several groups,

institutions or organisations may have made distinct research contributions to a given impact, and

strongly advise submitting institutions to ensure that both their own contribution is specified clearly

and that the contributions of others are acknowledged.

318. There will be many cases where a researcher has moved to a different institution during the

period in which a body of research underpinning a case study was produced. Where this is the case,

the submitting institution should make clear that the research undertaken during the period the

researcher spent at that institution made a material and distinct contribution to the impact claimed.

70

Main Panels A and B supplementary criteria – indicators of quality for underpinning

research

319. Case studies must include references to one or more key research outputs that identify the

research produced by the submitted unit that underpinned the impact, and must provide evidence

of the quality of the research. Case studies should include references to any REF-eligible

output(s) as defined in the output glossary that will best enable the panels to determine that the

two-star threshold has been met. They should include additional indicators, a s appropriate, of the

quality of the underpinning research. The sub-panels will use the information prov ided in case

studies, and may review research and outputs referenced in Section 3, in order to be assured that

the quality threshold has been met.

Main Panels C and D supplementary criteria – i ndicators of quality for underpinning

research

320. Submitting units must ensure that each case study fulfils the threshold criteri on on research

quality (see ‘Guidance on submissions’, paragraph 319.b). A sample of the research should be

cited that is sufficient to identify clearly the body of work, or individual project that underpins the

claimed impact.

321. Main Panels C and D wish to emphasise that the term ‘underpi nning’ does not imply a

specific temporal or directly causal relationship to the associated impact and recognise that in

some cases the research associated with the impact may be carried out at the same time as the

impact, and that the nature of the relationship between the research and the impact may be

complex and non-linear.

322. Sub-panels do not expect to read the underpinning research output(s) as a matter of course

to establish that the threshold has been met. The submitting institution should aim, where

possible, to provide evidence of this quality level. Some of the indicators of such quality might be

(but are not restricted to):

 research outputs which have been through a rigorous peer-review process

 evidence of peer-reviewed funding

 reviews of outputs from authoritative sources

 prizes or awards made to individual research outputs cited in the underpinning research

 evidence that an output is a reference point for further research beyond the original

institution.

71

323. Not all indicators of quality will apply to all forms of research output . If no such indicators are

available then the sub-panel will review the item in question to satisfy itself that this meets the

quality threshold.

Preparing impact case studies

324. The sub-panels recommend that institutions refer to the following list of characteristics when

preparing case studies:

 All the material required to make a judgement should be included in the case study template

(REF3) – no further reading should be required. URLs should only be inclu ded for the

purpose of verifying or corroborating claims made in the submission. Panels will not follow

URLs to access additional evidence or information to supplement the submission.

 There should be a clear definition of the beneficiaries, and w hat has changed as a result of

the research.

 The narrative should be coherent, clearly explaining the relationship between the

researchers, the underpinning research, the impact, and the nature of the changes or

benefits arising (noting that narratives differ according to the areas of impact claimed).

 Indicators used should be relevant, contextualised and precise in support of the case study,

and the evidence should be verifiable, focused and concise.

 There should be a brief explanation of what is original or distinctive about the research

insights that contributed to the impact.

 Specific and appropriate sources of corroborating information, independent o f the submitting

HEI, should be supplied. Extracts from corroborating statements may be included with in the

case studies, where appropriate.

 Where the research was carried out in collaboration with other HEIs, or was part of a wider

body of research, this should be acknowledged and the specific contribution to the impact of

the submitting unit’s research clearly described. In such cases, units (whether within or

across HEIs) may provide common descriptions of the impact arising, where they so wish.

72

Section 5: Environment

Environment criteria

325. The sub-panels will assess the environment according to the generic criteria and level

definitions in ‘Guidance on submissions’, Annex A. The main panels have set out below how the

criteria will be understood by their sub-panels.

326. Vitality will be understood as the extent to which a unit support s a thriving and inclusive

research culture for all staff and research students, that is based on a clearly articulated strategy for

research and enabling its impact, is engaged with the national and international rese arch and user

communities and is able to attract excellent postgraduate and postdoctoral research ers.

327. Sustainability will be understood as the extent to which the research environment ensures

the future health, diversity, wellbeing and wider contribution of the unit and the discip line(s),

including investment in people and in infrastructure.

328. In assessing the environment element of submissions, panels will assess vitality and

sustainability in terms appropriate to the scale and diversity of the research activity the submitting

unit supports, and as appropriate for its subject area(s). They will assess vitality and sustainability in

terms of both the research environment within the submitting unit, and its participation in and

contribution to its subject discipline, academic community and wider society.

329. In forming the environment sub-profiles, the sub-panels will attac h weightings to each of the

four sections of the unit-level environment template (REF5b), as set out below:

 unit context and structure, research and impact strategy

 people

 income, infrastructure and facilities

 collaboration and contribution to the research base, economy and society.

The assessment will take account of the environment data as stated in paragraphs 362 to 363 and

the information provided in the institutional-level statement (REF5a) as stated in paragraph 333.

Main Panels A, B and C supplementary criteria – section weightings for the environment

template (REF5b)

330. The sub-panels in Main Panels A, B and C will attach equal weighting to each of the four

sections of the unit-level environment template (REF5b).

73

Main Panel D supplementary criteria – section weightings for the environm ent template

(REF5b)

331. In view of the primary role that people play as the key resource in the arts and humanities,

the sub-panels in Main Panel D will attach differential weight to each of the components of the

environment template as follows:

Unit context and structure, research and impact strategy

25%

People

30%

Income, infrastructure and facilities

20%

Collaboration and contribution to the research base,

economy and society

25%

Institutional-level environment template (REF5a)

332. The following information is required in the institutional-level env ironment statement (REF5a):

a. Context and mission: an overview of the size, structure and mission of the institution.

b. Strategy: the institution’s strategy for research and enablin g impact (including integrity,

open research, considerations of equality and diversity, and structures to support

interdisciplinary research, where applicable) in the assessment period and for the next

five-year period.

c. People: the institution’s staffing strategy, support and training of research students, and

building on the information provided in codes of practice, evidence about how equa lity

and diversity in research careers is supported and promoted across the institution.

d. Income, infrastructure and facilities: the institutional-level resourc es and facilities

available to support research. This should include mechanisms for supporting the

reproducibility of research as appropriate to the research focus of the HEI, and to

facilitate its impact.

333. The sub-panels will use the information provided in the institutional-level state ment to inform

and contextualise their assessment of the relevant sections of the unit-level temp late. The

institutional-level statement will not be separately assessed or separately scored by the sub-panels.

Units should not repeat material covered in REF5a in REF5b and should cross-refer b etween the

statements, where appropriate.

74

334. As set out in ‘Guidance on submissions’ (paragraph 362), the REF5a statements will be

reviewed by a pilot assessment panel. The pilot exercise will consider the standalone assessment of

a discrete institutional-level environment element in future exercises, and will run concurrently to the

REF 2021 assessment.

335. Small and specialist institutions that will make a submission in one UOA only are encouraged

to submit a REF5a statement but are not required to do so. Where an HEI does not provide a REF5a

statement, the pilot panel will review the submitted REF5b template. In such cases, institut ions

should ensure that sufficient information is provided in the REF5b template about the institution’s

context and should be guided by the supplementary guidance to be provided by the pilot

assessment panel in summer 2019. Additionally, where there is any distinction between the research

and impact strategies, policies, facilities and resources between the institution and the submitting

unit, this should be clearly identified in the REF5b template.

Unit-level environment template (REF5b)

336. The main panels believe that excellent research can be undertaken in a wide variety of

research structures and environments, and outstanding impacts achieved from within a wide variety

of research contexts and resulting from a wide diversity of approaches. The main panels consider

that the health of the disciplines represented within the sub-panels is well served by this variety. The

main panels have no pre-formed view of the ideal size or organisational structure for a research

environment, or of the ideal context or approach for enabling impact, and will judge each submission

on its merits, contextualised appropriately to the nature of institution. Panels will assess, on an equal

basis, submissions that reflect the work of administrative units such as department s, and

submissions that do not map neatly onto departmental or other administrative struct ures within HEIs.

337. Given that there is no expectation that the environment element of submissions relates to a

single coherent organisational unit, submissions should explain any distinct groups or units covered,

particularly where discrete organisational units form part of a single submissio n.

338. Neither the existence of research groups, nor their absence, is, in itself, considered significant

by the sub-panels.

Use of indicators

339. As indicated in paragraph 362, all sub-panels will receive informat ion on doctoral degrees

awarded and research income. In the main panel supplementary criteria set out for the different

sections of the REF5b template, the panels indicate where additional data should be provided in the

template.

340. Further quantitative indicators may be included in REF5b, where appropriate, to support

claims made in the narrative. In identifying additional indicators for inclusion, sub mitting units are

strongly advised to refer to the advice and examples based on work carried out by the Forum for

Responsible Research Metrics, available at www.ref.ac.uk, under Guidance . However, the main

75

panels wish to make it clear that the examples should not be regarded as mandatory nor a ‘check-

list’ of additional requirements.

341. In assessing the environment template, sub-panels will take a holistic view of each section,

taking into account both the narrative and any supporting evidence provided. Where appropr iate,

panels will consider data in the context of size and type of institution in order to enable judgements

to be made on an equitable basis.

Template requirements

342. The main and sub-panels expect to see the unit’s approach to equality and diversity reflected

and evidenced as appropriate throughout the template below.

343. All the material required to make a judgement should be included in the environment template

– no further reading should be required. URLs should only be included for the purpose of verifying or

corroborating claims made in the submission. Panels will not follow URLs to access additional

evidence or information to supplement the submission.

344. Submitting institutions are reminded that the word limits for REF5a and REF5b (set out in

Annex F of the ‘Guidance on submissions’) are upper limits, not a minimum requirem ent.

345. The following information is requested for each of the sections in the environment template

(REF5b):

REF5b, Section 1: Unit context and structure, research and impact strategy

346. This section should provide evidence of the achievement of strategic ai ms for research and

impact during the assessment period, and details of future strategic aims and goals for research and

impact; how these relate to the structure of the unit, and how they will be taken forward. Note that

there is no expectation that this section refers to a single department or coherent organisational unit.

Evidence may include (but is not limited to):

 How research is structured across the submitted unit (including research groups or sub-

units), to provide context for assessing the submission.

 The submitting unit’s research objectives during the assess ment period and over the next

five years, including a review of the submitting unit’s research plans describ ed in REF 2014.

 How the unit has sought to enable and/or facilitate the achieveme nt of impact arising from

their research and how they are shaping and adapting their plans to ensure that t hey

continue to support the vitality and sustainability of the unit’s impact in the future. The

submitting unit should describe how the selected case studies relate to their approach t o

achieving impact.

76

 The submitting unit’s approach to supporting interdisciplinary research, where applicable, in

the context of the unit’s research strategy. The sub-panels will give due cre dit where these

arrangements have enhanced the vitality and sustainability of the research environment.

Units where this is not applicable will not be disadvantaged in the assessment.

 Within the context of the institution’s strategy, how t he submitting unit is progressing towards

an open research environment, including where this goes above and beyond the REF open

access policy requirements, and wider activity to encourage the effective sharing and

management of research data, as appropriate to the discipline. Consideration of

reproducibility should also be included where relevant to the discipline.

 Within the context of the institution’s approach, how t he unit supports a culture of research

integrity, and ensures that research is conducted according to appropriate ethical, lega l and

professional frameworks, obligations and standards.

Main Panel D supplementary criteria – REF5b: Unit context and structure, research and

impact strategy

347. Main Panel D is aware that the development of an open research environ ment poses

specific issues for the disciplines covered by its sub-panels. In this context, ev idence of an open

research environment could include, but is not limited to:

 any contribution from the unit to open access debates, processes and structures within the

subjects covered by the Main Panel

 any contribution from the unit to the development of an open research culture within the

subjects covered by the Main Panel, especially where these reflect on the distinctive

nature of research and research data as generated by the relevant subject communities

(e.g. IP and licensing constraints/issues). This could include, for example, a range of

mechanisms to share research and research data openly, as appropriate to the discipline.

REF5b, Section 2: People

348. This section should provide evidence about: staffing strategy and staff development within the

submitted unit, including evidence of how the staffing strategy relates to the unit’s research and

impact strategy and physical infrastructure; support for early career researchers and career

development at all stages in research careers; support mechanisms for, and evidence of the training

and supervision of, postgraduate research (PGR) students; and evidence of how the submitting unit

supports and promotes equality and diversity. Particular attention will be paid to how submitting units

address all relevant aspects of support for equality and diversity (which should be taken to refer to all

protected characteristics) within their submissions. There should be synergy between the strategies

77

and structures set out in this section and the institution’s code of practice, which the panels will be

able to access on request.

Staffing strategy and staff development

349. This may include (but is not limited to):

 staff development strategy for all staff pursuing a career in research at all stages of their

careers, including the use of mentoring, probation and appraisal and training, and the unit’s

implementation of the Concordat to Support the Career Development of Researchers

 the unit’s staffing and recruitment policy and evidence of its effectiveness, including the

pattern of staff recruitment over the assessment period, the balance between s hort-term and

long-term contracts among Category A eligible staff, and how the demogra phic profile of the

unit affects current and future management of research activity, including succession

planning

 evidence of how individuals at the beginning of their research career s are being supported

and integrated into the research culture of the submitting unit, including the contribution o f

postdoctoral researchers to the unit (where appropriate)

 the policy for research, impact leave/sabbatical leave for all staff at all stages of their careers

(including fixed-term and part-time staff)

 evidence of procedures to stimulate and facilitate exchanges b etween academia and

business, industry or public or third sector bodies, for example, through the recruit ment or

secondment of research staff

 how the unit recognises and rewards staff for carrying out research and for achieving impact,

and how the unit specifically supports and enables staff to achieve impact from the ir

research.

Research students

350. Submitting units are invited to provide evidence of the quality of training and supervision of

PGR students and how the unit has developed a research culture into which research students are

fully integrated and prepared for further research activity. This may include (but is not limited to):

 the approach to recruitment of doctoral research students, includi ng those with protected

characteristics

 evidence of studentships from major funding bodies (the sub-panels recognise the

challenges of recruiting doctoral students in the current funding environment)

 details of monitoring and support mechanisms linked t o evidence of progress and of

successful completions

78

 details of the support provided to research students in terms of skills development and

preparation for their future career.

Equality and diversity

351. Submitting units are invited to provide evidence of their commitment to equality and diversity in

the recruitment and support of staff with significant responsibility for research and research students,

including the strategies, activities and collaborations that support equality and diversity and enable

staff and research students drawn from a wide cross-section of society to engage in research. All

relevant protected characteristics should be considered, and the submitting unit may wish to include

(but is not limited to) evidence of:

 study leave arrangements (including supporting data where relevant)

 arrangements for supporting flexible and/or remote working

 the career pathways for part-time and fixed-term staff

 how conference attendance or other necessary travel to support research is facilitated for

staff and research students with caring responsibilities, ill health etc.

 how equality and diversity considerations are taken into account with regard to support for

submission of funding applications, access to internal funds, research-related promotion and

reward procedures, recruitment for research-related leadership roles, conference

attendance, sabbaticals and training

 support for staff and research students returning from periods of leave (including parental

leave) or ill health, managing long-term illness, or with caring responsibilities

 support for staff with protected characteristics (e.g. disabilities) to enable them to research

productively

 the submitting unit’s approach to supporting the wellbeing o f its staff and research students.

352. The submitting unit should also demonstrate how it has paid due regard to equality and

diversity issues in the construction of its REF submission, including the selection of the output

portfolio, and how data on the distribution of outputs across staff relate to the unit’s approach to

supporting equality and diversity. Units should explain how this approach relates t o the processes

set out in their institution’s code of practice.

Main Panel A supplementary criteria – REF5b: People

353. Where relevant, submitting units may also wish to include details of:

 effective integration of clinical academics and NHS-employed activ e researchers

79

 research career development of both non-clinical and clinical researc hers

 role of clinical researchers.

Main Panel C supplementary criteria – REF5b: People

354. All sub-panels in Main Panel C recognise the role of professional an d other doctoral

qualifications and their contribution to the vitality of the research environment. To obtain a clear

understanding of the nature of the research environment, units submitting in UOAs in Main Panel

C are asked to disaggregate the total number of doctoral degrees awarded as r eported in REF4a

for each year in the assessment period into PhDs and research-based professional doctorat es.

This information should be included as part of the ‘People: research students’ section of the

REF5b template. The disaggregated data should be presented in tabular format, reported in

academic years according to the standard data in section REF4a. The total disaggre gated data

should sum to the totals reported in REF4a.

REF5b, Section 3: Income, infrastructure and facilities

355. This section should provide information about the income, infrastru cture and facilities

pertaining to research and research impact, including but not limited to:

 research funding and strategies for generating research income, including that allocated as

part of larger research consortia, links between research funding and high-quality researc h

output or impact, and major and prestigious grant awards made by external bodies on a

competitive basis. Allowance will be made for disciplines that find it more difficult to attract

research funding because of the nature of the research, and where more early career

researchers are involved

 organisational infrastructure supporting research and impact , for example, evidence of areas

where there has been significant investment, or through the development of research

clusters that focus on distinctive areas of work, which may include the delivery of highly

impactful research

 operational and scholarly infrastructure supporting research and impact within the submitting

unit, including technical and support staff, estate and facilities, advanced equipment, IT

resources or significant archives and collections

 how any relevant equality and diversity issues have been addressed, for example in relation

to support for acquiring research funding, or support for accessing scholarly or operational

infrastructure

 how infrastructure, facilities and expertise are utilised in relation to impact activities

80

 the nature, quality, provision and operation of specialist research infrastructure and facilities

 evidence of cross-HEI shared or collaborative use of research infrastructure including the

use of major research facilities both in the UK and overseas

 significance of major benefits-in-kind (including, for example, donated items of equipment,

sponsorships secured, or other arrangements directly related to research).

Main Panel B supplementary criteria – REF5b: Income, infrastructure and facilities

356. For Sub-panels 8 and 9 only: data should be prov ided on usage within the assessment

period (1 August 2013 to 31 July 2020) of major national and international facil ities not supported

by the Research Councils which was awarded to an investigator in the submitted unit aft er

competitive review by a pa nel of internationally recognised exp erts. The information should be

provided for each facility in terms of the time awarded together with t he total cost, where av ailable.

Main Panel D supplementary criteria – REF5b: Income, infrastructure and facilities

357. Submissions should, where possible, detail funding that has been received through sources

not reported in Higher Education Statistics Agency (HESA) returns, such as commissio ns from

artistic organisations and other sources, including from overseas, and how these relate to the

research activities, outputs and/or impact of the submitting unit.

REF5b, Section 4: Collaboration and contribution to the research base, economy and society

358. This section should provide information about collaboration and contribut ion, including:

 the arrangements, support in place for and effectiveness o f research collaborations,

networks and partnerships, including joint research projects with academic colleagues in

other institutions, locally, nationally or internationally and indicators of their success

 evidence of how staff interacted with, engaged with or developed relationsh ips with key

research users, beneficiaries or audiences in the period 2014 to 2020 to develop impact from

the research carried out in the unit and how these collaborations have enriched the research

environment

 wider contributions to the economy and society, including evidence of the wider activities and

impact of research carried out in the unit that is not captured in the impact case studies

 how the unit engages with diverse communities and publics through its research

81

 evidence of the unit’s contribution to the sustainability of the discipline, support for and

exemplars of interdisciplinary research, and responsiveness to national and international

priorities and initiatives

 indicators of wider influence, contributions to and recognition by the research base including,

but not limited to:

o journal editorship

o participation on grants committees

o fellowships

o prizes

o membership of Research Council or similar national and internationa l committees

o invited keynotes, lectures and/or performances, or conference chair roles

o refereeing academic publications or research proposals

o co-operation and collaborative arrangements for PGR training, including whe ther these

have received formal recognition nationally or internationally.

Main Panel A supplementary criteria – REF5b: Collaboration and contributi on to the

research base, economy and society

359. Where applicable, submitting units should identify the number of staff meeting the definition

of Category C staff (‘Guidance on submissions’, paragraphs 374 to 376 ) on the census date, 31

July 2020, and describe their contribution to the research environment and, where relevant, the

unit’s submission.

360. Submitting units should also provide evidence of:

 the approach to encouraging and developing best practice in undertaking research that is

reproducible, including any papers that are reproducing key papers in the field

 the extent of collaboration or integration with external organisations such as health or

social care services (e.g. NHS and social care structures) and/or with industry or

government bodies, where appropriate.

Main Panel D supplementary criteria – REF5b: Collaboration and contrib ution to the

research base, economy and society

82

361. Main Panel D would prefer to see Section 4 of the template constructed as a narrative

rather than a series of lists.

Environment data (REF4a/b/c)

362. ‘Guidance on submissions’ (Part 3, Section 4) sets out quantitative data relating to the

research environment to be included in all submissions (REF4a/b/c). Sub-panels will use the data in

the context of the information provided in the environment template (REF5b) to inform their

assessment. Data on research doctoral degrees awarded (REF4a) will be used to inform the sub-

panels’ assessment in relation to ‘research students’ (Section 2: People). D ata on research income

(REF4b/c) will be used to inform the sub-panels’ assessment in relation to ‘Section 3: Income,

infrastructure and facilities’.

363. Data on both doctoral degrees awarded and research income will be considered in the context

of the narrative provided in the REF5b template, and taking account of the size of the submitting

unit, its areas of specialism, its research groups, research strategy and different lev els of research

funding available in different fields.

364. The sub-panels do not require quantitative data provided by institutions in REF4a/b/c to be

reported by research group.

83

Part 4: Panel procedures

Panel competence to do business

365. Each main and sub-panel will consider, confirm and document its competen ce to do business

at the start of each assessment meeting, taking into consideration the range of expertise as well as

the numbers of panel members present.

366. Where there is a foreseen absence of a sub-panel chair at a main panel meeting, the main

panel chair will consider whether it requires the attendance of the deputy sub-panel chair in order to

be competent to do business. Attendance of the deputy sub-panel chair at main panel meetings will

only be allowed in this case, and at the discretion of the main panel chair.

Dealing with absences of the chair

367. Each main and sub-panel will elect a deputy chair for planned and unfore seen absences of the

chair, and in cases where there is a major conflict of interest for the chair. In the absence of the

chair, the deputy will chair meetings of the panel. Where both the chair and deputy declare a conflict

of interest in the same institution, the panel will nominate one of the remaining mem bers to officiate

in that instance.

Conflicts of interest

368. All REF main and sub-panel chairs, members, assessors, observers, secretaries and advisers

will observe the arrangements for managing potential conflicts of interest set out in Annex D.

Confidentiality arrangements

369. All REF main and sub-panel chairs, members, assessors, secretaries, advisers and observers

are bound by the terms of the REF confidentiality arrangements as detailed in Annex E. These

arrangements have been put in place to enable the effective management and operation of the REF,

and for the protection of panel members.

Data protection

370. All REF main and sub-panel chairs, members, assessors, secretaries, advisers and observers

shall ensure that personal data are kept securely, maintained confidentially and used only for the

purposes set out in the Guidance on Submissions and the Panel Criteria and Worki ng Methods. All

personal data are subject to the provisions of the Data Protection Act 2018 (DPA2018) and the

General Data Protection Regulation (GDPR), and all persons storing, handlin g or processing

personal data for the REF shall adhere to the principles and requirements set out in the DPA2018

and the GDPR.

84

Part 5: Panel working methods

Main panel working methods

371. Each main panel has worked with its sub-panels to review and define commo n assessment

criteria, as set out above. Main panels will work with their sub-panels throughout the assessment

process to ensure that the published procedures are followed and that the overall asses sment

standards are applied consistently. Each main panel will also be responsible for deciding on the

quality profile to be awarded to each submission in each of the UOAs in its remit, after

recommendations have been made by the sub-panels.

372. Each main panel will work with its sub-panels as follows:

a. Main panel meetings. The main panels will meet regularly throughout the planning and

assessment phases to ensure close working and communication between sub-panels,

to identify issues for early action, seek advice on handling specific cases, resolve

emerging differences, share developing good practice and provide assurance on the

procedures being followed. Sub-panel chairs will report to the main panel meetings on

general progress and on the implementation of working methods, particularly o n issues

where cross-panel consistency is significant, including:

 interdisciplinary research outputs

 cross-referrals

 the range of output types

 impact case studies

 double-weighted outputs.

b. Main panel member attendance at sub-panel meetings. The chair and members of

the main panel will attend some meetings of sub-panels, to provide assurance that

practices are consistent across the group of sub-panels:

 The members of the main panel advising on interdisciplinary research will in

particular be engaged in calibration processes relating to interdisciplinary research

outputs, supporting the interdisciplinary advisers on the sub-panels, and advising on

the consistency of assessment standards for interdisciplinary research.

 The international members of the main panel will, in particular, be engaged in sub-

panel calibration processes and in the formation of quality profiles, to ensure

consistency with international standards.

 Main panel user members will, in particular, be engaged in briefing and calibration

among sub-panel user members and assessors, providing support and focus for

85

them, and advising on consistency of method and efficient use of expertise and

knowledge in assessing impact case studies.

 The main panel chair and main panel members will attend a sample of sub-panel

meetings as agreed with the main panel, especially at an early stage in the

assessment process.

c. Advice and support to panels. A group of panel advisers and panel secretaries will be

appointed to support the work of each main panel and its sub-panels. The secret ariat

will be briefed and trained in providing advice and guidance to their group of panels on

the assessment procedures. Each member of the panel secretariat will work with several

sub-panels within a main panel, providing consistent support and advice across them

and providing feedback to the main panel chairs as appropriate.

d. Cross-panel appointments. Individual academic members and assessors, and

individual user members or assessors, may be appointed to work with more than one

sub-panel, particularly where there are substantial overlaps between UOAs, to

contribute to consistency in the assessment of work on the boundaries. In consider ing

the selection and appointment of further members and assessors, t he main panel will

identify where such boundaries could benefit from joint appointments.

e. Calibration exercises. Each main panel and its sub-panels w ill undertake calibration

exercises at an early stage in the assessment to develop a common understanding o f

the assessment standards and the application of the quality levels. Internation al and

user members of the main panel will participate in these exercises to assist in

benchmarking judgements. The main panel chair and members of the main panel will

attend a selection of the sub-panel meetings that deal with calibration exercises, and

main panels will receive and discuss reports from sub-panel chairs on these exercises.

f. Reviewing emerging assessment outcomes. The main panels will review the

emerging assessments at UOA level from their sub-panels during the course of the

assessment phase, to support the consistent application of assessment standards. To

facilitate this review, the group of sub-panels within each main panel w ill adopt a

common process for the formation of each of the three sub-profiles and a common

sequence in which each sub-profile will be formed. In considering the emerging

assessment outcomes from sub-panels, the main panels will seek advice from the

international members about the application of internationally referenced stan dards, and

from the user members about the assessment of impact.

g. Deciding on the outcomes. When cons idering the quality profiles recommended by its

sub-panels, each main panel will confirm that the published assessment procedures and

criteria have been applied by the sub-panels, and that the sub-panels have consistently

applied the overall standards of assessment. The main panels recognise that there may

be a range of overall profiles across their respective UOAs reflecting the relativ e

86

strength of the disciplines in the UK. Each main panel will require that any substantia l

differences in the overall profiles for each UOA are investigated and understood be fore

approving the quality profiles recommended by its sub-panels. Where the

recommendations of a given sub-panel for the overall results for that UOA are at

substantial variance from the other sub-panels, the sub-panel chair will nee d to justify

this to the main panel with reference to external evidence where available.

373. In addition to the main panels’ approaches to ensuring consiste ncy within each group of sub-

panels, to support appropriate consistency across the four main panels:

a. Generic assessment criteria and working methods across all main and sub-panels have

been developed, as set out in ‘Guidance on submissions’ and throughout t his document.

These include standard weightings for each of the elements of the assessment (output s,

impact and environment), generic criteria for assessing each element, a consistent

approach to staff circumstances, and consistent working methods and procedures.

b. The four main panel chairs, the REF director and the panel advisers will meet regularly

throughout the assessment phase to discuss progress, identify issues for early action

and inform the work of the main panels. This will include planning and reporting on

calibration exercises, including cross-main panel calibration, and reviewing emerging

and final outcomes across the four main panels. Specific actions will be identified t o

support consistency across those sub-panels in different main panels that have a

significant overlap (for example, sharing of some of the material used in calibration

exercises, and identifying opportunities for appointing assessors to work across those

sub-panels).

Sub-panel working methods

374. Each sub-panel will be responsible for assessing submissions in its UOA, applying the

published criteria and working methods, and recommending the outcomes to the main panel. This

section sets out how the sub-panels will undertake their work at each stage of the assessment

process.

Appointing the full assessment phase sub-panel

375. In early 2020, the sub-panels will examine institut ions’ submission intentions and identify their

requirements for further members and assessors to ensure the sub-panel has the breadth and depth

of expertise required to carry out the assessment. These members and asses sors will be appointed

prior to the start of the assessment to ensure that sub-panels have access to appropriate expertise

to reach robust and valid judgements with regard to the material anticipated in sub missions. The

procedures for appointing members and assessors are described in paragraphs 27 to 36.

376. Sub-panels will consider the breadth of work in actual submissions early in the assessment

phase in 2021 in order to confirm that the sub-panel collectively has the breadth and depth of

 
87 
 
 
expertise to assess the work submitted. Where necessary,                        sub-panels may recommend the 
appointment of further additional assessors or, exceptionally, request that
                        speci
                        fic parts of 
submissions should be cross-referred to another sub-panel (as described in
                        paragraphs 399 to 404 
and indicated, where appropriate, in the UOA descriptors). 
377.  Each sub-panel will include research user members and impact assessors,
                         with appropriate 
                    
expertise to contribute fully to the assessment of the impact element of submis
                        sions, alongside 
academic members of the sub-panels. The research user members and impact
                        assessors will be 
                    
appropriately briefed (for example, with respect to the details of the REF process
                        and key issues, 
including equality and diversity) alongside the sub-panel members and output                        assessors.  
                    
Allocating work 
378.  The sub-panel chair, consulting with the deputy chair, interdiscip
                        linary adviser(s) and sub-
                    
panel members, as appropriate, will allocate work to members and assessors
                        w
                        ith appropriate 
expertise, taking account of any conflicts of interest and the sub-panel’s approach
                        to identifying 
                    
outputs with significant material in common (see paragraphs 214 to 215). Thi
                        s allocation may be at 
the level of individual or groups of outputs, individual or groups of impact
                        case studies, and whole 
environment templates. 
379.  Each member and assessor on a sub-panel will be allocated a significant
                        volu
                        me of material to 
                    
assess, so that each member and assessor makes a significant contribution t
                        o the sub-panel’s 
overall recommendations. 
380.  Each impact case study will be allocated to at least one academic
                        member and one user 
                    
member or assessor, wherever practicable. User assessors will be allocated impact
                        cas
                        e studies, 
and may be allocated relevant parts of the environment template. User members
                        may
                         – in addition 
to impact case studies – be allocated whole environment templates and/or out
                        puts in particular 
areas where they are willing and have appropriate expertise to assess them.
                    
381.  Where a sub-panel cross-refers parts of a submission to another                        sub-panel for advice, the 
                    
procedures in paragraphs 399 to 404 will be followed. W
                        here a sub-panel refers outputs in a 
                        
                    
language other than English to external specialist advisers, the procedures in
                        ‘G
                        uidance on 
                        
                    
submissions’ paragraphs 285 to 287 will be followed.
                        
                        
                        
                    
Calibration of assessment standards 
382.  Sub-panels will undertake early calibration exercises with
                        respect to outputs, impact and 
                    
environment, to ensure sub-panel members and assessors develop a common
                         understanding of the 
                        
                    
quality levels. The calibration exercises will be based on samples of a range
                        of outputs (whether 
submitted to the REF or sourced from elsewhere by panel members), on samples
                        of submitted 
impact case studies and environment templates.  
383.  In addition to sub-panel members, the assessors who will
                        subsequently
                         be involved in 
                    
assessing either outputs or impact will take part in the relevant calibration
                        exercises.
                         

88

384. After these initial calibration exercises, the sub-panels will continue to discuss the application

of the quality levels and will keep under review the scoring patterns of member s and assessors, to

ensure consistency in the sub-panel’s standards of assessment.

Assessing submissions

385. Sub-panels will assess all of the components of submissions: research outputs, impact and

the research environment. This reflects an underpinning principle that sub-panels will assess each

submission in the round. They will not make collective judgements about the contributions of

individual researchers. Sub-panels will make collective judgements about the range of submitted

information in order to develop the sub-profiles and recommend the overall quality profile, for each

unit being assessed.

386. All the outputs listed in submissions will (unless prevented by reasons beyond a sub-panel’s

control) be examined by panel members and/or assessors. They will be examined w ith a level of

detail sufficient to contribute to the formation of a robust sub-profile for all the outputs in that

submission. In doing so panels will take into account additional information where relevant (as

described above), but expert review of the outputs will remain the primary means of assessing them.

387. Sub-panels will examine all the submitted case studies, and all the information submitted in

the environment template together with the standard data analysis.

388. Sub-panels will meet during the course of the assessment phase to discuss their assessment

of each element of submissions. Assessors will attend those meetings at which the relevant element

of submissions is being discussed, so that they contribute fully and on an equal basis with members,

to the development of the relevant sub-profile.

389. During the course of the assessment, the sub-panels will be asked to draw attention to any

data they would like the REF team to verify through an audit. These data will be investigated by the

REF team (in addition to the REF team auditing a proportion of submitted information from each

institution, as described in ‘Guidance on submissions’, paragraphs 91 to 97).

Assessing interdisciplinary work

390. For the purposes of the REF, interdisciplinary research is understood t o achieve outcomes

(including new approaches) that could not be achieved within the framework of a single discipline.

Interdisciplinary research features significant interaction between two or more discipli nes and/or

moves beyond established disciplinary foundations in applying or integrating researc h approaches

from other disciplines.

391. Submitting HEIs should identify those outputs that they consider meet the definition of

interdisciplinary research, as set out in ‘Guidance on submissions’ (paragraph 273) and repeated in

this document in paragraph 390, to draw this to the attention of the sub-panel. Outputs flagged as

interdisciplinary will be assessed on a fair and equal basis and will be neither advantaged nor

disadvantaged in the assessment. Sub-panels may identify outputs as interdisciplinary that have not

 
89 
 
 
been flagged as such by the submitting HEI, to enable consideration of the most
                        appropr
                        iate means 
of assessing the output. Additionally, there will be no disadvantage in the assessme
                        nt where a sub-
panel considers that a flagged output does not meet the definition of interdisciplinary
                         research and 
the output will be assessed on a fair and equal basis with other submitted
                        outputs. In assessing 
interdisciplinary outputs, sub-panels will make use of the criteria referred to in paragraph
                        19
                        6. 
392.  The interdisciplinary identifier will allow panels, working with their
                        IDR advisers, to consider the 
                    
most appropriate means of assessing the output. It is distinct from the cross-referral
                        proc
                        ess, which 
is set out below in paragraphs 399 to 404. Flagging
                        an output as interdisciplinary will not trigger 
cross-referral of the output; cross-referral may be one of the assessment routes
                        fo
                        llowed. The 
panels do not anticipate that all interdisciplinary outputs will require cross-referral
                        and, conversely, 
expect that cross-referred outputs will not all necessarily meet the REF definition
                        of interdisciplinary 
research. 
393.  Given the anticipated diversity of both the interdisciplinary                        submissions and the sub-panels, it 
                    
is not appropriate to prescribe a single approach to assessing interdisciplinary                        outputs. The 
processes will be developed by the individual sub-panels and will be tailored
                        to the submissions they 
receive. However, there are a number of mechanisms in place to ensure that
                        interdi
                        sciplinary 
outputs are assessed consistently across the panels, as set out in paragraphs
                        396 to 398. 
394.  Each sub-panel will have members who have experience of
                        interdisciplinary
                         work. Where 
                    
appropriate, this expertise will be augmented with the appointment of additional
                        members and 
assessors. Sub-panels are confident that they can assess such work, and the
                        appointment of the full 
membership for the assessment phase will seek to ensure that sub-panels have
                        access to 
appropriate expertise to reach robust and valid judgements with regard to submitted
                        material. 
395.  All sub-panels will have at least two members appointed as interdisciplinary
                        advisers. The 
                    
interdisciplinary advisers will offer guidance to the sub-panels in their assessment
                        of interdisciplinary 
outputs to enable their robust and valid assessment. This may include advising
                        on the allocation of 
outputs and the calibration and moderation of scoring. Interdisciplinary advisers
                        wil
                        l not necessarily 
be expected to assess all interdisciplinary outputs submitted to their
                        panel. 
                    
396.  The interdisciplinary advisers will work in a network with their
                        counterpart
                        s on other sub-panels 
                    
(across all four main panels) to review outcomes from the initial calibration of
                        interdiscip
                        linary 
outputs and may be involved in the joint consideration of outputs with other advisers
                        fro
                        m the 
relevant sub-panels, as required during the assessment process. The network
                        of interdisciplinary 
advisers will meet at key points during the assessment phase, which will provide
                        a forum for 
reviewing joint working arrangements and identifying wider expertise requirements.
                        IDAP will provide 
advice and support to the network on these aspects during the calibration and
                        assess
                        ment phases. 
397.  The sub-panels’ approach to assessing outputs identified as
                        interdisciplinary will be reviewed 
                        
                    
across the main panels, to ensure an appropriate consistency of approach. An
                        alysis of the emerging 
and final scores for the group of outputs identified as interdisciplinary will
                        be conducted by IDAP 
during and upon completion of the assessment process. 

90

398. IDAP will have a role in overseeing the assessment of interdisciplin ary work to ensure that

agreed principles and process for assessment are applied and that there is consiste ncy in approach

across panels. IDAP will not advise on the assessment of individual outputs, but will advise on

process, and provide advice and support for cross-panel calibration. The main panel interdisciplinary

leads will join the membership of IDAP during the assessment phase.

Cross-referral of parts of submissions

399. The sub-panels’ preferred approach is to assess work within the sub-panel to which it was

submitted and, informed by the survey of submission intentions, to appoint further members and

assessors where required to enable this. In cases where, in the sub-panel’s opinion , the sub-panel

and its appointed assessors do not have the required expertise to assess specific p arts of

submissions, those parts of submissions may be cross-referred to other sub-panels for advice.

400. The submitting HEI may request that specific parts of submissions shou ld be cross-referred to

another sub-panel for advice. The sub-panels will consider such requests and decide up on the most

appropriate means of assessing the material in question:

a. Where the sub-panel considers there is sufficient expertise within the sub-panel to reach

a robust judgement, the work will be assessed within the sub-panel. The sub-panels

expect that this will normally be the case, except where the UOA descriptors indicate

specific arrangements for cross-referral.

b. In those instances where the sub-panel does not consider it contains the appropriate

expertise, it may cross-refer the work to an appropriate sub-panel for advice (whether

within or outside the same main panel). The REF director will work with the main panels

to ensure consistency of approach in cross-referring work across the sub-panels.

401. In addition to considering requests made by institutions, sub-panels may identify specific parts

of submissions that it considers should be cross-referred to another sub-panel, and request that

such work should be cross-referred.

402. The sub-panels’ approach to cross-referral will be discussed within the main panels, to ensure

an appropriate consistency of approach.

403. Entire submissions may not be cross-referred. Specific outputs may be cross-referred. The

original sub-panel will specify the scope of advice that it is seeking. This will be limited to advice

relating to the quality of outputs. It may not include advice on other matters such as the contribution

of a co-author or double-weighting of outputs. In exceptional cases, sub-panels may cross-refer

impact case studies for advice (including advice related to the quality threshold for the underpinning

research).

404. Where parts of submissions are cross-referred, advice will be sought and given on the basis of

any specific assessment criteria and procedures (including, for example, use or not of citation data)

for the UOA in which the work was originally submitted; cross-referred parts of submissio ns will be

91

assessed on the same basis as work which is not cross-referred. The original sub-pane l will retain

responsibility for recommending the quality profile for all work that was submitted in its UOA.

Developing and recommending quality profiles

405. Sub-panels will develop a sub-profile for each of the three element s – outputs, impact and

environment – of each submission.

406. Outputs sub-profile. Each output listed in a submission w ill be assessed against the quality

levels: 4*, 3*, 2*, 1* or ‘unclassified’. The outputs sub-profile will be formed by calculating the

percentage of outputs listed in a submission that are assigned at each quality level, with each output

contributing an equal proportion to the sub-profile. The following exceptions and rules apply:

a. Any submitted output that is found to be ineligible will be entered into t he outputs sub-

profile as ’unclassified’.

b. Where a submitted member of staff is found to be ineligible, that member of staff and the

outputs attributed to them will be removed from the submission; those outputs w ill not

contribute to the outputs sub-profile (see ‘Guidance on submissions’, par agraphs 136 to

137).

c. Any outputs that are ‘missing’ from a submission (that is, where fewer outputs have

been submitted than the number required, as specified in paragraph 205 of ‘Guidance

on submissions’, and where no staff circumstances apply), will be graded as

‘unclassified’.

d. Where a request to double-weight an output has been accepted by the sub-panel, the

quality level assigned to the output will be entered twice into the outputs sub-profile.

Where a request to double-weight an output is not accepted by the sub-panel the

reserve output will be assessed. If no reserve output has been submitted, the output will

contribute to the sub-profile as a single output and one instance of ‘unclas sified’ will be

entered into the outputs sub-profile (‘Guidance on submissions’, paragraph 281).

e. Where the sub-panel determines that the submitted member of staff against whom a co-

authored output is listed did not make a substantial contribution to the output, the output

will be entered into the outputs sub-profile as ‘unclassified’ (paragraph 219).

f. Where a sub-panel judges that two outputs within a submission do not contain

sufficiently distinct material and should be treated as a single output, an unclassified

score will be given to the ‘missing’ output (paragraph 214).

407. Impact sub-profile. Each case study included in a submission will be assessed according to

the definitions of the starred levels in ‘Guidance on submissions’ (Annex A). Any case studies that

are ‘missing’ from a submission (that is, where fewer case studies have been submitted than the

number required, as specified in paragraphs 309 to 310 of ‘Guidance on submissions’) will be

92

graded as ‘unclassified’. The impact sub-profile will be formed by calculating t he percentage of

impact case studies listed in a submission that are assigned at each quality level, with each impact

case study contributing an equal proportion to the sub-profile.

408. Environment sub-profile: Sub-panels will assess the in formation provided in the environment

template (REF5b), and consider the environment data within the context of that information. Sub-

panels will build up a graduated sub-profile by assessing the range of elements in each submission,

using the starred levels defined in ‘Guidance on submissions’ (Annex A). In the environment criteria

definitions, the main panels indicate the weighting that the sub-panels will atta ch to each component

of the environment template.

409. The three sub-profiles will be combined into an overall quality profile, using the weightings and

method described in ‘Guidance on submissions’ (Annex B).

410. In recommending the overall quality profile for each submission to its main panel:

a. Each sub-panel will reach a collective decision, within the framework of the exercise and

in accordance with the published statement of criteria and working methods. E ach sub-

panel will debate the reasoning behind the quality profiles in sufficient detail to reach

collective conclusions, and will make recommendations to the main panel on the basis

of its collective judgement. Each sub-panel will seek to achieve a consensus on all the

overall quality profiles to be recommended to its main panel. If a consensus cannot be

achieved after reasonable effort, decisions will be taken by majority vote, with the chair

holding a casting vote.

b. Each sub-panel will confirm to the main panel that each submission has been assessed

against the published criteria for that UOA (including in cases where parts of

submissions have been cross-referred to other sub-panels for advice) and according to

the published procedures.

c. Each sub-panel will confirm that each submission has been examined in s ufficient detail

to form robust judgements, and that appropriate expertise has been deployed in

assessing submissions.

Recording panel decisions

411. The panel secretariat will minute details of the procedures followed by panels, and these will

be published after the conclusion of the exercise. Panels will not make or record collective

judgements about individuals’ contributions to submissions. The panel secretariat w ill record the

panels’ collective judgements about the sub-profiles and overall quality profiles in respect of each

submission.

93

Annex A: Examples of impacts and indicators

1. Table 1 is intended to illustrate the wide variety of areas in which impact from research across the panels may be found to have a

positive influence on the quality of life of individuals and communities locally, national ly and internationally. These are indicative only, and

in practice much of the impact will cross boundaries between them or go beyond t hem. Case studies are not expected to be classified in

this way by submitting units. A searchable database of impact case studies su bmitted to REF 2014 can be found here:

http://impact.ref.ac.uk/CaseStudies/Search1.aspx.

2. The ‘indicators’ are listed independently of the ‘types of impact’ and are not intended to link to a specific impact example listed. The

list provides illustrative examples of indicators of both reach and significance. The panels set out their approach to assessing impact

against these criteria in Part 3, Section 3, paragraphs 287 to 324.

3. Examples of impact achieved through public engagement are integrated into the different areas of impact in Table 1. More detailed

advice on achieving and evidencing impact through public engagement can be found on the website of the National Coordinating Centre for

Public Engagement: http://www.publicengagement.ac.uk/.

4. Examples are also provided of impact evaluation frameworks used outside higher education. Impact partners may also have their

own evaluation frameworks that could be drawn upon to evidence impact.

Table 1

Areas of impact

Types of impacts

Indicators of reach and significance

Impacts on the health and wellbeing

of people, and animal welfare

Impacts where the beneficiaries are

individuals and groups (both human and

animals) whose health outcomes have

been improved, whose quality of life has

been enhanced (or potential harm

mitigated) or whose rights or interests

 Outcomes for patients/users or related groups

have improved.

 Public health or wellbeing has improved.

 Quality of life in a developing country has been

improved by new products or processes.

 A new clinical or lifestyle intervention (e.g. drug,

diet, treatment or therapy) has been developed,

 Measures of improved clinical outcomes,

public behaviour or health services (lives

saved, reduced infection rates).

 Measures of improved wellbeing.

 Evidence from clinical trials.

94

have been protected or advocated

through the application of enhanced

policy and practice for individuals or

public health activities.

trialled with patients/users, related or other groups

(e.g. community samples), and definitive (positive

or negative) outcome demonstrated.

 Patient health outcomes have improved through,

for example, the availability of new drug, treatment

or therapy, diagnostic or medical technology,

changes to patient care practices, or changes to

clinical or healthcare guidelines.

 A new diagnostic or clinical technology has been

adopted.

 Disease prevention or markers of health have

been enhanced by research.

 Misleading health claims identified by research

are not included in food packaging.

 Care and educational practices have changed.

 Clinical, dietary, health or social care guidelines

have changed.

 Health or social care training guidelines have

changed.

 Decisions by a health service or regulatory

authority (to take, or not to take action) have been

informed by research.

 Measures of improved patient/user

outcomes, public health or health services.

 Documented changes to clinical and/or

public health guidelines (documented

references to research evidence in

guidelines).

 Evidence of enhancement of patient/user

experience.

 Evidence of take-up and use of new or

improved products and processes that

improve quality of life or animal welfare in

any given context, e.g. developing countries.

 Evidence of the number of animals no longer

used in research or a specific sector (e.g.

per test, drug, laboratory, or leisure

industry).

 Documented changes to animal welfare

codes or guidelines.

95

 Public health and quality of life has been

enhanced through, for example, enhanced public

awareness of a health risk, enhanced disease

prevention or, in developing countries, improved

water quality or access to health and social care.

 The user experience has improved.

 Increased patient/user involvement in shaping and

implementing policy and practice.

 Public awareness of a health risk or benefit has

been raised.

 The control of diseases has changed in

developing countries.

 Development or adoption of new indicators of

health and wellbeing.

 Development of policy and practice with regard to

medical ethics, health services or social care

provision.

 Influence on CPD and training standards.

 Influence or shaping of relevant legislation.

 Influencing policy or practice leading to improved

take-up or use of services.

 Improved provision or access to services.

96

 Animal health and welfare has been enhanced by

research.

 Use of animals in research has been reduced,

refined or replaced.

Impacts on creativity, culture and

society

Impacts where the beneficiaries may

include individuals, groups of

individuals, organisations or

communities whose behaviours,

creative practices, rights, duties and

other activity have been influenced.

 Collaboration with museum professionals results

in enhancements to (cultural) heritage

preservation and interpretation, including museum

and gallery exhibitions.

 Co-production of new cultural artefacts, including

for example, films, novels and TV programmes.

 Generating new ways of thinking that influence

creative practice, its artistic quality or its audience

reach.

 Inspiring, co-creating and supporting new forms of

artistic, literary, linguistic, social, economic,

religious, and other expression.

 Collaboration with public arts venues, artists and

programming professionals to produce new forms

of artistic expression.

 Research-led engagement with marginalised,

under-engaged and/or diverse audiences leads to

increased cultural participation.

Arts Council England offer guidance and toolkits

for evaluating impact:

 https://www.artscouncil.org.uk/quality-

metrics/quality-principles

 https://www.artscouncil.org.uk/quality-

metrics/quality-metrics

 https://www.artscouncil.org.uk/measuring-

outcomes/generic-learning-outcomes

 https://www.artscouncil.org.uk/measuring-

outcomes/generic-social-outcomes

 Testimonials from creative practitioners,

curators, media professionals.

 Publication and sales figures both in the UK

and overseas, audience or attendance

figures (including demographic data where

relevant), broadcasting data and other forms

97

 Developing stimuli to cultural tourism and

contributing to the quality of the tourist experience.

 Improvements to legal and other frameworks for

securing intellectual property rights.

 Increased understanding of local traditions leads

to enhanced cultural preservation in any given

context, for example developing countries.

 New forms of artistic expression resulting in

enhancement of quality of life.

of media, download figures, or database and

website hits over a sustained period.

 Evaluative reviews in the media.

 Citations in reviews outside academic

literature. Independent citations in the

media, including in online documents.

 Tourism data, including audience figures and

visitor numbers at exhibitions, events,

performances.

 Professional evaluations of exhibitions,

performances or other outputs.

 Audience/visitor/participant feedback (e.g.

through surveys, interviews or focus groups).

Impact on social welfare

Impacts where the beneficiaries include

individuals, groups of individuals,

organisations or communities whose

rights, duties, behaviours, opportunities,

inclusion, quality of life and other activity

have been influenced.

 Improved social welfare, equality, social inclusion;

improved access to justice and other opportunities

(including employment and education).

 Engagement with research has enhanced policy

and practice for securing poverty alleviation.

 Influential contributions to campaigns for social,

economic, political and/or legal change through

engagement with civil society groups.

A beginner’s guide to evaluating social return on

investment (SROI) can be found here:

http://www.socialvalueuk.org/resource/guidance-

on-starting-out-on-sroi-2/.

 Documented evidence of changes to social

policy.

 Measures of improved social equality,

welfare or inclusion.

98

 Changes to social policy have been informed by

research.

 Changes to social policy have led to improved

social welfare, equality or social inclusion.

 Research has contributed to community

regeneration or development.

 Improved social and educational inclusion of

marginalised groups in any given context, for

example developing countries.

 More effective integration of refugees into host

communities.

 Enhanced understanding of victims’ needs in

reconciliation processes in post-conflict states.

 Citations in campaign literature (e.g.

leaflets).

 Evidence of public debate in the media or

other fora being influenced by the research.

 Documented evidence of increased social

inclusion (e.g. participation figures).

 Testimonials from civil society groups and

policymakers.

Impacts on commerce and the

economy

Impacts where the beneficiaries may

include businesses, either new or

established, the NHS, private health

and social care, agriculture or other

types of organisation which undertake

activity that may create wealth.

 A spin-out or new business has been created,

established its viability, or generated revenue or

profits.

 Contributing to innovation and entrepreneurial

activity through the design and delivery of new

products or services.

 Decisions are made not to introduce a new

process or product as a result of research.

 Social enterprise initiatives have been created.

 Evidence of improved cost-effectiveness.

 Evidence of service change.

 Sales of new products/services.

 Business performance measures (e.g.

turnover/profits, trends in key technical

performance measures underlying economic

performance).

 Employment figures.

99

 The costs of treatment, health or social care have

changed as a result of research-led changes in

practice.

 Policies have been introduced which have had an

impact on economic growth or incentivising

productivity.

 Gains in productivity have been realised as a

result of research-led changes in practice.

 Research helps to stimulate foreign direct

investment (FDI).

 The performance of an existing business has

been improved through the introduction of new, or

the improvement of existing, products, processes

or services; the adoption of new, updated or

enhanced technical standards and/or protocols; or

the enhancement of strategy, operations or

management practices.

 Contributing to economic prosperity via the

creative sector including publishing, music,

theatre, museums and galleries, film and

television, fashion, tourism, and computer games.

 Performance has been improved, or new or

changed technologies or processes adopted, in

companies or other organisations through highly

skilled people having taken up specialist roles that

draw on their research, or through the provision of

 Licences awarded and brought to market;

market authorisation.

 Demonstrable collaborations with industry

(including knowledge transfer partnerships,

and contracts).

 Commercial adoption of a new technology,

process, knowledge or concept.

 Business performance measures, for

example sales, turnover, profits or

employment associated with new or

improved products, processes or services.

 Jobs created or protected.

 Investment funding raised from UK and/or

non-UK agencies (venture capital/Business

Angel, and so on) for start-up businesses

and new activities of existing businesses.

 Priority shifts in expenditure profiles or

quantifiable reallocation of corporate, non-

profit or public budgets.

 Evidence of critical impact on particular

projects, products and processes confirmed

by independent authoritative evidence,

which should be financial where possible.

100

consultancy or training that draws on their

research.

 Potential future losses have been mitigated by

improved methods of risk assessment and

management in safety- or security-critical

situations.

 The strategy, operations or workplace practices of

a business have changed.

 Improved support for the development of ‘small

scale’ technologies.

 Improvements in legal frameworks, regulatory

environment or governance of business entities.

 Better access to finance opportunities.

 Enhanced corporate social responsibility policies.

 More effective dispute resolution.

 Alternative economic models (such as fair trade)

have been developed and adopted.

 Evidence of research leading to avoidance

of negative outcomes.

 Quantitative data relating, for example, to

cost-effectiveness or organisational

performance.

 Tourism data, including audience figures and

visitor numbers at exhibitions, events,

performances.

 Evidence of closing identified skills gaps.

Impacts on public policy, law and

services

Impacts where the beneficiaries are

usually government, non-governmental

organisations (NGOs), charities and

 Policy debate has been stimulated or informed by

research evidence, which may have led to

confirmation of policy, change in policy direction,

implementation or withdrawal of policy.

 Documented evidence of use in policy

debate (e.g. at a parliamentary Select

Committee, material produced by NGOs).

 Citation in a public discussion, consultation

document or judgement.

101

public sector organisations and society,

either as a whole or groups of

individuals in society, through the

implementation or non-implementation

of policies, systems or reforms.

 Policy decisions or changes to legislation,

regulations or guidelines have been informed by

research evidence.

 A policy has been implemented (including those

realised through changes to legislation) or the

delivery of a public service has changed.

 In delivering a public service, a new technology or

process has been adopted or an existing

technology or process improved.

 The quality, accessibility, acceptability or cost-

effectiveness of a public service has been

improved.

 (Sections of) the public have benefited from public

service improvements.

 Risks to the security of nation states have been

reduced.

 The work of an NGO, charitable or other

organisation has been influenced by the research.

 Legislative change, development of legal principle

or effect on legal practice.

 Research is used by parliamentarians to develop

proposals for new legislation through Private

Members’ Bills, or to assist scrutiny of legislation

 Evidence of citation in policy, regulatory,

strategy, practice or other documents.

 Direct citations of research in parliamentary

publications such as Hansard, committee

reports, evidence submissions, or briefings.

 Acknowledgements to researchers on

webpages, in reports or briefings.

 Evidence of influence on a debate in public

policy and practice through membership of

or distinctive contributions to expert panels

and policy committees or advice to

government (at local, national or

international level).

 Quantitative indicators or statistics on the

numbers of attendees or participants at a

research event, or website analytics for

online briefings.

 Qualitative feedback from participants or

attendees at research events.

 Data to show close working relationships

with members or staff. For example, the

number of meetings held, minutes from

these meetings, membership of working

groups, co-authoring of publications.

102

and inform amendments to other bills such as

those introduced by government.

 Research recommendations are taken up by

policymakers through membership of a

government advisory committee.

 Policymakers make use of research-based critical

evidence synthesis in developing policy.

 Government analysts adopt innovative

methodological or approach-based advice from

researchers.

 Forms of regulation, dispute resolution or access

to justice have been influenced.

 Research is used to change current processes or

services, or identify new services to be provided.

 Research into the languages and cultures of

minority linguistic, ethnic, religious, immigrant,

cultures and communities used by government,

NGOs, charities or private sector to understand

and respond to their needs.

 Research helps to highlight issues of concern to

parliamentarians and contributes to new analysis

of existing issues.

 Research helps parliamentarians and staff to

identify inquiry topics, shape the focus of inquiries,

 Testimonials from members, committees or

officials, where available.

 Documented evidence of influence on

guidelines, legislation, regulation, policy or

standards.

 Documented evidence of changes to public

policy, legislation, regulations or guidelines.

 Analysis by third-party organisations of

parliamentary proceedings or processes, for

example studies of the passage of particular

pieces of legislation.

 Documented evidence of changes to

international development policies.

 Evidence of use of process/technology.

 Measures of improved public services,

including, where appropriate, quantitative

information; such information may relate, for

example, to the quality, accessibility or cost-

effectiveness of public services.

 Measures of improved inclusion, welfare or

equality.

 Satisfaction measures (e.g. with services).

103

inform questioning of witnesses, and underpin

recommendations.

 Research equips parliamentarians, their staff, and

legislative staff with new analytical or technical

skills, or refreshes existing ones.

 International policy development has been

influenced by research.

 Allocation and/or distribution of Official

Development Assistance (ODA) has been

influenced by research.

 Policy and practice of international agencies or

institutions have been influenced by research.

 Research stimulates critical public debate that

leads to the non-adoption of policy.

 Formal partnership agreements or research

collaboration with major institutions, NGOs

and public bodies. Consultancies to public or

other bodies that utilise research expertise.

 Evidence of engagement with campaign and

pressure groups and other civil organisations

(including membership and activities of

those organisations and campaigns) as a

result of research.

 Documented evidence of changes to

international development policies.

 Measures of improved international equality,

food security, welfare or inclusion.

Impacts on production

Impacts where the beneficiaries are

individuals (including groups of

individuals) whose production has been

enhanced.

 Production, yields or quality have been enhanced

or level of waste has been reduced.

 Research helps to create routes to international

innovation and market impact.

 Research leads to improvement in productivity

and resource-use efficiency.

 A new product has been recommended for

use or adopted.

 Development of a new plant variety or crop

protection product which has entered the

appropriate national or international

regulatory testing system.

 Evidence of improved sustainability.

104

 Decisions by regulatory authorities have been

influenced by research.

 More efficient production, including food

production, for example where costs have been

reduced.

 Animal husbandry methods have changed.

 Management practices in production businesses

have changed.

 Documented changes to working guidelines.

 Documented evidence of improved working

practices and/or level of production.

Impacts on practitioners and delivery

of professional services, enhanced

performance or ethical practice

Impacts where beneficiaries may

include organisations or individuals,

including service users, involved in the

development and/or delivery of

professional services and ethics.

 Professional standards, guidelines or training

have been influenced by research.

 Professional methods, ideas or ethics have been

influenced by research.

 Professionals and organisations are able to adapt

to changing cultural values as a result of research.

 Contribution to continuing personal and

professional development.

 Practitioners/professionals/lawyers have used

research findings in conducting their work.

 Professional bodies and learned societies have

used research to define best practice, formulate

 Documented change to professional

standards, performance or behaviour.

 Evidence of adoption of best practice (e.g.

by educators or law enforcement personnel).

 New or modified professional standards and

codes of practice.

 New or modified technical standards or

protocols.

 Documented changes in knowledge,

capability or behaviours of individuals

benefiting from training.

105

policy, or to lobby government or other

stakeholders.

 Workforce planning has been influenced by

research.

 Educational or pedagogical practices and

methods have changed in primary, secondary,

further or higher education, within or beyond the

submitting unit.

 Practices have changed, or new or improved

processes or methods have been adopted, by

individuals, companies or other organisations,

through the provision of training or consultancy.

 The development of expert systems has been

influenced in areas such as medicine, human

resources, accounting, and financial services.

 The quality, efficiency or productivity of a

professional service has improved.

 Expert and legal work or forensic methods have

been informed by research.

 Law enforcement and security practices have

changed.

 Cessation of practices shown by research to be

ineffective.

 Evidence of debate among practitioners,

leading to developments in attitudes or

behaviours.

 Literature/web information from practitioners

and advisers, including the research findings

and how they are applied in practice.

 Traceable reference to inclusion of research

in national or international industry standards

or authoritative guidance.

 Traceable references by practitioners to

research papers that describe their use and

the impact of the research.

106

Impacts on the environment

Impacts where the key beneficiaries are

the natural, historical and/or built

environment, together with societies,

individuals or groups of individuals who

benefit as a result.

 The environment has been improved through the

introduction of new product(s), process(es) or

service(s); the improvement of existing product(s),

process(es) or services; or the enhancement of

strategy, operations or management practices.

 New methods, models, monitoring or techniques

have been developed that have led to changes or

benefits.

 Policy debate on climate change or the

environment has been influenced by research.

 Policy debate on the environment, environmental

policy decisions or planning decisions have been

stimulated or informed by research and research

evidence.

 Improved design or implementation of

environmental policy or regulation.

 The management or conservation of natural

resources, including energy, water and food, has

changed in a developing country.

 The management of an environmental risk or

hazard has changed.

 Changes in environmental or architectural design

standards or general practice.

 Sales of new products, or improvements in

existing products, that bring quantifiable

environmental benefits.

 Verifiable influence on particular projects or

processes which bring environmental

benefits.

 Evidence of generic environmental impact

across a sector, confirmed by independent

authoritative evidence.

 Traceable reference to inclusion of research

into government policy papers, legislation

and industry guidance.

 Traceable reference to the influence of

research in planning decision outcomes.

 Sales of new products or improvements in

existing products that bring quantifiable

environmental benefits.

 Traceable impacts on particular projects or

processes which bring environmental

benefits.

 Evidence of generic environmental impact

across a sector, confirmed by independent

authoritative evidence.

107

 Influence on professional practice or codes.

 Changes in practices or policies affecting

biodiversity.

 The operations of a business or public service

have been changed to achieve environmental

(green) objectives.

 Direct intervention, based on research evidence,

has led to a reduction in carbon dioxide or other

environmentally damaging emissions.

 Increased understanding of the environmental

impact of a product or process means that it is not

adopted by industry.

 Documented case-specific improvements to

environment-related issues.

 Traceable reference to inclusion of research

into government policy papers, legislation

and industry guidance.

 Traceable reference to impact of research in

planning decision outcomes.

Impacts on understanding, learning

and participation

Impacts where the beneficiaries are

individuals, communities and

organisations whose awareness,

understanding, participation or

engagement have been enhanced as a

result of research.

 Enhanced cultural understanding of issues and

phenomena; shaping or informing public attitudes

and values.

 Public interest and engagement in research has

been stimulated through, for example, the

enhancement of science education in schools.

 The awareness, attitudes or understanding of

(sections of) the public have been informed, and

their ability to make informed decisions on issues

improved, by engaging them with research.

 Public or political debate has been shaped or

informed by research; this may include activity

Many organisations use the Generic Learning

Outcomes (GLO) to evaluate impacts on

knowledge and understanding:

https://www.artscouncil.org.uk/measuring-

outcomes/generic-learning-outcomes.

The Heritage Lottery Fund also offers guidance

on evaluating participation and learning:

https://www.hlf.org.uk/evaluation-guidance.

108

that has challenged established norms, modes of

thought or practices.

 Contributing to processes of commemoration,

memorialisation and reconciliation.

 Contributing to a wider public understanding of

basic standards of wellbeing and human rights

conceptions.

 Contributing to widening public access to and

participation in the political process.

 Professionals and organisations have adapted to

changing cultural values.

 Research has challenged conventional wisdom,

stimulating debate among stakeholders.

 Increased understanding of gender roles in any

given context (e.g. developing countries) has

improved equality.

 Changes to education or the school curriculum

have been informed by research.

 Influencing the design and delivery of curriculum

and syllabi in schools, HEIs or other educational

institutions.

 Documented evidence that public

understanding has been enhanced through

active collaborative involvement in research.

 Documented evidence of policy debate (e.g.

in Parliament, the media, material produced

by NGOs).

 Public debate in the media.

 Documented shift in public attitude (e.g. to

sexual behaviour, or social factors in health).

 Documented evidence of enhanced

awareness of health risks and benefits by

consumers.

 Citation in a public discussion, consultation

document or judgement.

 Citation by journalists, broadcasters or social

media.

 Evidence of increased public uptake of

scientific training, through public

engagement.

 Information about the number and profile of

people engaged and types of audience.

109

 Research results in changes to the delivery of

vocational courses and subsequently changes to

professional practice.

 Reduced gap in academic attainment for students

with protected characteristics.

 Evidence of secondary reach, for example

from follow-up activity or media coverage.

 Evidence of sustainability through, for

example, a sustained or ongoing

engagement with a group, a significant

increase in participation in events or

programmes or use of resources.

 Evidence of engagement with campaign and

pressure groups and other civil organisations

(including membership and activities of

those organisations and campaigns) as a

result of research.

 Measures of increased attainment and/or

measures of improved engagement with

science in non-HE education.

 Evidence of use of education materials

arising from the research.

110

Annex B: Summary of additional information about outputs

1. This annex provides a summary table of all the additional information statem ents relating to

outputs that are required in submissions (in form REF2). It should be read alongside, and does not

replace, the guidance provided in ‘Guidance on submissions’ and in the relevant parts of the panel

criteria statements, as indicated in ‘Summary of additional information required about outputs’ below.

It is intended for institutions’ ease of reference in identifying the requirements for additional types of

information about outputs, across the four main panels.

2. The word limits for the additional information about outputs are common across the four main

panels, as set out in ‘Word limits for additional information about outputs’ bel ow.

Summary of additional information required about outputs

Main Panel A

Main Panel B

Main Panel C

Main Panel D

a. Outputs that include significant material published prior to 1 January 2014 (paragraphs 214

to 215)

Panel

requirements:

All main panels: Statement on how far the earlier work was revised to incorporate

new material

Reference:

Part 3, Section 2, paragraph 215

b. The researcher’s contribution to a co-authored or co-produced output (paragraphs 216 to

220)

Panel

requirements:

Affirmation of the

author’s

contribution to the

output (selected

from the

statements

provided) only

where the author

is not the lead or

corresponding

author and the

output has 15 or

more co-authors

Sub-panels 7, 8,

10, 11 and 12:

None

Sub-panel 9:

Affirmation of the

author’s

contribution to the

output only where

the author is not

the lead or

corresponding

author and the

output has 15 or

more co-authors

None

111

Reference:

Part 3, Section 2,

paragraphs 221 to

225

Part 3, Section 2,

paragraph 226 to

231

Part 3, Section 2,

paragraph 231 to

232

Part 3, Section 2,

paragraph 233 to

236

c. Request to double-weight an output (paragraphs 237 to 241)

Panel

requirements:

All main panels: a supporting statement to justify the request

Reference:

Part 3, Section 2, paragraphs 242 to 243

Part 3, Section 2, paragraphs 244 to 247

d. Abstract for outputs in languages other than English (paragraph 272)

Panel

requirements:

All main panels: For all outputs in languages other than English, a short abstract t o

describe the nature and content of the work. (This requirement is waived for outputs

submitted in UOA 26 (Modern Languages and Linguistics) in a language included in

the sub-panel descriptor – see Part 3, Section 2, paragraph 272.)

Reference:

Part 3, Section 2, paragraph 272.

e. Information about the research process and/or content

Panel

requirements:

Statement where

this is not evident

within the output

(for non-text or

practice-based

outputs)

Statement where

this is not evident

within the output

(for non-text or

practice-based

outputs)

Identification of the

original research or

new insights

reported (for

reviews)

Statement where

this is not evident

from the output

itself (for any type

of output)

For practice-based

outputs, an

explanatory

presentation in

paper format

should be included

For software and

data sets, a full

written description

with details how to

access

Statement for any

output where the

role of the

researcher, or

research process,

is not evident

within the output

Statement on the

contribution of the

attributed author to

anthologies, edited

books, special

issues of journals

and curatorial

projects

Rationale for

grouping short

items as a single

output

112

Reference:

Part 3, Section 2,

paragraph 252

Part 3, Section 2,

paragraphs 254 to

255

Part 3, Section 2,

paragraphs 259 to

262

Part 3, Section 2,

paragraphs 263 to

273

f. Factual information about the significance of the output

Panel

requirements:

None

In UOAs 11, 12,

Factual statement

wherever available

None in UOAs 7, 8,

9 and 10

None

Reference:

Part 3, Section 2,

paragraph 253

Part 3, Section 2,

paragraphs 256 to

257

n/a

Word limits for additional information about outputs

Type of information

Word limit

a. Outputs that include significant material published prior to 1

January 2014

Maximum 100 words for each of

a–d

b. The researcher’s contribution to a co-authored or co-

produced output

c. Request to double-weight an output

d. Abstract for outputs in languages other than English

e. Information about the research process and/or content

Maximum 300 words

f. Factual information about the significance of the output

Maximum of 100 words

113

Annex C: Main Panel D – output types & submission guidance

1. Sub-panels in Main Panel D receive the widest diversity of output types across the exercise. Each submitted output needs to have a

single classification selected from this list. The purpose of the classification is to assist in the management of the collection and distribution

of outputs, the allocation of outputs to reviewers and a post-submission analysis of types of outputs submitted. The sub-panel will assess

the research content of the material submitted regardless of the classification.

2. The format for the presentation of outputs (irrespective of their classificat ion) is flexible, within the requirement to limit the format

either to an electronic submission which is submitted via the REF submission sy stem either as a URL, DOI, or by uploading a PDF; or as

physical material which is sent to the REF team, and which may include digital/ electronic material on a media storage device e.g. USB. An

individual output cannot be submitted both electronically via the REF submission sy stem and as a physical output.

3. It is the responsibility of the submitting HEI to ensure that any digital material submitted is accessible from a range of devices.

4. Please also cross-refer to Part 3, Section 2, paragraphs 263 to 271, and Annex B, covering ‘Additional Information’.

Type

Recommend

ed additional

statements

(making

clear the

research

process/

content/contr

ibution)

Medium